PROTEINS: Structure, Function, and Genetics 52:51– 67 (2003)

Assessment of Blind Predictions of Protein–Protein

Interactions: Current Status of Docking Methods
Raúl Méndez,1,2 Raphaël Leplae,1 Leonardo De Maria,1,3 and Shoshana J. Wodak1*
1
Service de Conformation de Macromolecules Biologiques, et Bioinformatique, Centre de Biologie Structurale et
Bioinformatique, CP 263, BC6, Université Libre de Bruxelles, Bruxelles, Belgium
2
Grup de Modelització Estructural i Funcional de Sistemes Biològics, Institut de Neurociències, Unitat de Bioestadı́stica,
Facultat de Medicina, Universitat Autonòma de Barcelona Bellaterra, Spain
3
Centro Internacional de Fisica, A.A. 4948, Bogota DC, Colombia

ABSTRACT The current status of docking pro- Key words: structure predictions; residue-residue
cedures for predicting protein–protein interactions contacts; interface residues
starting from their three-dimensional structure is
assessed from a first major evaluation of blind pre- INTRODUCTION
dictions. This evaluation was performed as part of a
Intermolecular interactions, particularly those between
communitywide experiment on Critical Assessment
proteins, have become a central focus in postgenomic
of PRedicted Interactions (CAPRI). Seven newly
biology.1– 8 Tens of thousands of gene products are known
determined structures of protein–protein complexes or suspected to interact with many others, from genetic,
were available as targets for this experiment. These biochemical, or bioinformatics studies forming millions of
were the complexes between a kinase and its pro- putative complexes.9 –12 However, a very small fraction of
tein substrate, between a T-cell receptor ␤-chain these complexes will in the near future be available for
and a superantigen, and five antigen-antibody com- analysis, let alone for crystallization. Therefore, predicted
plexes. For each target, the predictors were given modes of association between macromolecules should be a
the experimental structures of the free components, useful guide for genetic and biochemical experiments, but
or of one free and one bound component in a ran- the prediction methods, commonly called docking algo-
dom orientation. The structure of the complex was rithms, must first be extensively tested and their validity
revealed only at the time of the evaluation. A total of assessed.
465 predictions submitted by 19 groups were evalu- Docking algorithms, first suggested in 1978,13,14 operate
ated. These groups used a wide range of algorithms on the atomic coordinates of two individual proteins usu-
and scoring functions, some of which were com- ally considered as rigid bodies and generate a large
pletely novel. The quality of the predicted interac- number of candidate association modes between them.
tions was evaluated by comparing residue–residue These candidates are then ranked by using various crite-
contacts and interface residues to those in the X-ray ria, used independently or in combination. The criteria
structures and by analyzing the fit of the ligand generally include geometric and chemical complementar-
molecules (the smaller of the two proteins in the ity measures, electrostatics, H-bonding interactions and
complex) or of interface residues only, in the pre- solvation, and use empirical potentials or database-
dicted versus target complexes. A total of 14 groups derived functions. A number of algorithms and many
produced predictions, ranking from acceptable to different scoring functions have been developed in the last
highly accurate for five of the seven targets. The use 10 years, as recently reviewed by several authors,2,15 and
of available biochemical and biological information, the field has become extremely active.
and in one instance structural information, played a Authors developing docking procedures usually test
key role in achieving this result. It was essential for them on crystal structures of complexes deposited in the
identifying the native binding modes for the five Protein Data Bank, which so far have been protease-
correctly predicted targets, including the kinase- inhibitor and antigen-antibody complexes. In the most
substrate complex where the enzyme changes con- favorable case, the best prediction is reasonably close to
formation on association. But it was also the cause
for missing the correct solution for the two remain-
Grant sponsor: Marie Curie Host Training; Grant number: QLK3-
ing unpredicted targets, which involve unexpected 1999-51297.
antigen-antibody binding modes. Overall, this anal- *Correspondence to: Shoshana J. Wodak. Service de Conformation
ysis reveals genuine progress in docking proce- de Macromolecules Biologiques, et Bioinformatique, Centre de Biolo-
dures but also illustrates the remaining serious gie Structurale et Bioinformatique, CP 263, BC6, Université Libre de
Bruxelles, Blvd. du Triomphe, 1050 Bruxelles, Belgium. E-mail:
limitations and points out the need for better scor- [email protected]
ing functions and more effective ways for handling Received 20 December 2002; Accepted 30 December 2002
conformational flexibility. Proteins 2003;52:51– 67.
© 2003 Wiley-Liss, Inc.

© 2003 WILEY-LISS, INC.

52 R. MÉNDEZ ET AL.

the crystal structure,16 –27 but none of the docking proce- cillus casei HprK, and its protein substrate Hpr (from B.
dures achieves this on all test complexes. Furthermore, subtilis).32 Targets T02–T06 were the following five differ-
the procedures perform poorly when the docking calcula- ent antigen-antibody complexes. The complex between the
tions use the unbound molecules, which often display some bovine rotavirus VP6 protein with Fab fragments from a
differences in conformation relative to the bound ones, monoclonal antibody (Rey et al., unpublished), the com-
either in their side-chains, or main-chain, or both. plex between the influenza hemagglutinin and an Fab
Previous attempts critically assessed docking proce- fragment that prevents the hemagglutinin low pH transi-
dures by comparing their performance in blind trials, in tion,33 and the complexes between porcine pancreatic
which the X-ray structures of the components are known ␣-amylase and three different variants of the single VHH
and that of the complex were made available only at the domain of camelid antibodies.34 Target 7 (T07) was the
time of evaluation.28 –30 But these attempts were limited complex between the mouse T-cell receptor ␤-chain and
in scope, with only two available targets and less than a the superantigen Streptoccoccal pyrogenic exotoxin A.35
handful of participating groups. The atomic coordinates of the targets were provided to
Here we present the results from an independent evalu- the assessors before publication to conduct the indepen-
ation of blind predictions of protein–protein interactions. dent evaluation reported here.
These predictions were performed as part of the Critical To perform the prediction for each target, predictors
Assessment of PRedicted Interactions (CAPRI), a commu- were given the atomic coordinates of the two components
nitywide experiment analogous to CASP,31 but aimed at to be used in their docking calculations. The 3D structures
assessing the performance of procedures for predicting the of these components, with the corresponding PDB codes
mode of association of two proteins based on their three- when applicable, are listed in Table I of Janin et al. (this
dimensional (3D) structure. issue). For the five antigen-antibody complexes, no 3D
Seven protein–protein complexes were available as tar- structures were available for the unbound Fab fragments.
gets for the first two rounds of CAPRI evaluated here. Therefore, the predictors were given the atomic coordi-
These comprised a complex between a kinase and its nates of the bound Fab molecules, but after their overall
protein substrate, that of a T-cell receptor ␤-chain and a orientation was randomized. This was not considered a
superantigen, and five antigen-antibody complexes. In serious bias given that antibodies generally undergo lim-
several of the complexes, one of the partners was a large ited conformational changes on antigen binding.
multimeric protein. The seven targets cover a range of biological systems
For each target, the experimental structures of the free and represent different degrees of difficulty. As seen from
components, or of one free and one bound component in a the Table I of Janin et al. (this issue), the interfaces in
random orientation, were made available to the predictors, these complexes bury surface areas in the range of 1200 –
but the structure of the complex was revealed only at the 2300 Å2, similar to those encountered in many protein–
time of the evaluation. protein complexes.36 These interfaces involve between 17
Nineteen groups submitted a total of 465 predicted and 34 residues on each protein, and those are engaged in
complexes, with not all the predictor groups submitting 37– 65 residue–residue contacts. The types of residues
predictions for every target. No limits were set on the involved in these interfaces (charged, polar, and hydropho-
source or type of additional information (homologue pro- bic) are also typical.
teins, biochemical data on interacting residues) that the The HprK-Hpr complex (T01) was clearly one of the
predictors could use to guide their docking calculations. most difficult problems. The free HprK protein is a hex-
The presented evaluation was performed with no knowl- amer and although the complex is known to involve one
edge of the identity of the predictors. It is based on a Hpr molecule per kinase subunit, the predictors had to
number of simple criteria, agreed on by the CAPRI manage- make a choice between using the entire hexamer, a trimer
ment team and approved by the participating groups only, or just the monomer, with the latter being a perilous
during the first CAPRI evaluation meeting held in France choice because it excluded the possibility for the Hpr
on September 19 –21, 2002 (see http://capri.ebi.ac.uk). The
molecule to simultaneously interact with two kinase sub-
results are presented in ways that should enable easy
units, as it is in fact the case.
perusal and comparison of the predictions (a set of simple
Because most docking programs are still limited in their
numbers and pictures), with the following main goals:
capacity in dealing with conformational adjustments, a
assessing the performance of available docking proce-
further difficulty arose from the fact that kinase conforma-
dures, identifying useful applications of available methods
tions in the free and bound forms differed. The main-chain
in structural genomics and proteomics efforts, and evaluat-
of individual bound and unbound monomers superimpose
ing the prospects of making progress in the field in the
with a root-mean-square deviation (RMSD) of 1.96 Å, with
near future.
a significant movement of the C-terminal helix (from
residue 287 on), which makes contact with the Hpr mole-
THE TARGET COMPLEXES AND THEIR
COMPONENTS cule in the complex. The monomers move moreover rela-
tive to one another.
The seven targets of this first CAPRI experiment are In addition, the structures of the unbound HprK mol-
denoted T01–T07 in Table I of Janin et al. (this issue). T01 ecules available in the PDB (see Table I in Janin et al.) had
was the complex between the protein kinase from Lactoba- missing coordinates for the loop comprising residues 241–
 BLIND PREDICTIONS OF PROTEIN–PROTEIN INTERACTIONS 53

252. Therefore, predictors needed to model the structure some more recent algorithms, they are directly incorpo-
for this loop, to avoid the risk of missing solutions that rated in the sampling step.
involve interactions with it. The ability of most procedures to handle conformational
However, there were some helpful clues as well. It was changes that may occur when the two proteins interact is
known that the phosphorylation site of the Hpr molecules still insufficient. Several allow for limited adjustments of
was Ser-46, and not His-15, and hence that the kinase the side-chains and backbone conformations by relaxation
P-loop needed to be close to this residue in the cognate via molecular mechanics during a refinement step or
complex. Ample information on amino acid sequences of account for flexibility by a built-in “softness” of the poten-
related kinases and Hpr molecules was also available. tial. However, some completely repack side-chains at the
The problem of conformational changes was negligible interface, whereas others dock sets of conformations from
for the remaining targets. The largest structural changes molecular dynamics simulations, rather than a single
between the unbound and bound molecules are for target structure, to mimic conformational flexibility.
T07 where the backbones of the variable domains (bound/ It is noteworthy that at least six procedures used in this
unbound) display an RMSD of 1.08 Å (Table I Janin et al.). first CAPRI challenge are novel or have novel aspects, and
However, the large size of the virus antigen moieties in many of these are still in the development stage. Hence,
targets T02 and T03 was a drawback for most docking their present performance is in no way indicative of their
programs. To deal with such large molecules, predictors intrinsic value. It should also be emphasized that the
often use common knowledge or biochemical information docking procedures differ significantly in calculation speed.
to limit the search space around regions of the proteins Some take minutes, whereas others take hours or days for
that are most likely to interact. For these targets such the same problem. However, comparing the speeds of
regions were the complementarity determining regions different procedures was not possible at this stage because
of the large variety of hardware and scoring functions
(CDRs) of the antibody and the portions of the antigen that
used.
would be accessible to an antibody on the virus envelope.
But restricting the search space bears the danger of THE EVALUATION PROTOCOL
missing the correct solution, as reported by some predic-
tors (Eisenstein). As seen below, the use of prior assump- To assess the quality of the predicted complexes, their
tions was a serious problem for the two simpler complexes atomic coordinates were first processed to correctly align
of the porcine ␣-amylase with the camelid VHH domains the various chain segments to their counterparts in the
(T04 and T05), where the antibodies bind in a rather targets. Procedures implemented in the BRUGEL pack-
unexpected mode involving a large number of framework age39 were then used to compute a set of parameters aimed
residues. at evaluating different aspects of the predicted complexes,
In terms of potentially helpful information, a low- which might be useful for different applications. For
resolution structural description37 was available for T02, example, accurate modeling of the predicted interface for
and a crystal structure of a related complex (1jck) with the use in designing inhibitors requires that residue–residue
same mode of interaction was available for the TCR␤-SpeA contacts and, if possible, atomic contacts be correctly
complex. identified, whereas information of the interface residues
might be sufficient for mutagenesis experiments looking
THE DOCKING PROCEDURES for mutants that inhibit the interaction. On the other
hand, fitting an atomic model into a low-resolution elec-
Table I gives a short summary of the docking procedures tron density map derived from electron microscopy40 is not
and the assumptions and constraints used by the different sensitive to the exact nature of the interacting residues
groups in predicting the CAPRI target complexes. but rather to the overall geometry of the complex (relative
A sizeable fraction of the procedures uses a cubic grid orientation and position of the receptor and ligand mol-
representation of the protein surface and fast Fourier ecules). In the following paragraphs, the parameters used
transform search algorithms, following earlier work. An- in the CAPRI evaluation are described in detail.
other involves alternative representations of the protein
models and different algorithms for sampling interaction Residue–Residue Contacts and Interface Residues
modes, which include geometric hashing, Monte Carlo, Two quantities pertaining to residue–residue contacts
genetic algorithm and Molecular, or Brownian-type, me- between the docked proteins were computed for each
chanics procedures. predicted complex. A pair of residues on different sides of
The criteria used to single out the correct interaction the interface was considered to be in contact if any of their
from the large number of incorrect solutions are also atoms were within 5 Å.
rather diverse. Some still rely essentially on purely geomet- One quantity is the fraction of native contacts fnat
ric criteria, but the more recent methods make increasing defined as the number of native (correct) residue–residue
use of composite scoring functions, which involve contribu- contacts in the predicted complex divided by the number of
tions from H-bonds, electrostatics, and solvation. How- contacts in the target complex. The other is the fraction of
ever, usually these contributions are severely approxi- non-native contacts fnon-nat, defined as the number of
mated and often used only to rescore a subset of solutions non-native (incorrect) residues–residue contacts in the
selected on the basis of geometric complementarity. But in predicted complex divided by the total number of contacts
54 R. MÉNDEZ ET AL.

TABLE I. Procedures for Protein–Protein Docking Used in CAPRI

Predictor Program Algorithm CAPRI specifics
Scripps, US (Abagyan) ICM-DISCO Rigid body pseudo-Brownian MC with grid-based T01: filtered for H15 or D46 in contact with phosphate
energy function; refinement: grid-based Biased and ranked according to score. T02–T06: filtered,
Probability MC Minimization using ICM for CDRs in contact; T07: filtered for VHH CDRs in
(internal coordinate mechanics). contact, and no clashes with TCR-␣.
Boston U. US (Camacho) SmoothDock Clustering, refinement and discrimination T01: restricted search to presumed phosphate binding
protocol: constrained minimization using region. T04: restricted to solutions involving CDRs.
CHARMM; Coulomb and solvation. T07: ranked first solution as in homologue.
Weizmann Inst. IL MolFit Weighted geometric docking with FFT followed T01: up-weight conserved residues in HPrK and HPr;
(Eisenstein) by clustering and filtering. Scoring by filtering for solutions with D46 directed to P-loop.
weighted geometric complementarity. T02–T06: up-weight interactions with CDRs; T04–
T06 filter against solutions with non-CDR contacts.
T07: homologue used, down-weight residues in
TCR␣/␤ contacts.
Cancer Research UK LRI Guided Docking Rigid body molecular mechanics using Only T04–T07 predicted. Partial manual ranking,
(Fitzjohn/Bates) CHARMM22 force field. Flexible refinement T07: homologue (1JCK)used
step in some cases. Solvation used in final
energy ranking.
Sheffield U. UK GAPDOCK GA for sampling different relative rotations and T01, T02 only. Search restricted to expected contact
(Gardiner) clustering. Scoring based on shape correlation regions. Final selection, manual.
and buried area, as per CCP4 package; some T01: selected solutions with Ser 46 contacting
clash checks. ligands on kinase. T02: limited search space, and
selected solution containing CDRs.
U. Washington US (Gray/ Monte Carlo and clustering, side-chain T01–T03: no minimization; T01: search restricted to
Baker) repacking, rigid body minimization. Fitted HPr. Ser46-HPrK. Asp157 in contact, kinase
multiterm scoring function, mainly vdW conserved residues. T02–T06: favored contacts
orientation-dependent hydrogen bonding; with CDRs; T02: only outer cap.; T07: used
implicit solvation. homologue, manual selection except for model1
Aberdeen U. UK CONCOORD43⫹Hex Dock 500 essential dynamics structures from T07: 500 SpeA structures docked onto rigid TCR.
(Mustard) CONCOORD using Hex. Search constrained to TCR hypervariable loops.
Columbia U. US (Norel) PPD Geometric hashing; multiple rescoring.
Scripps US Surfdock46 Fourier correlation of spherical harmonics. Only T01–T03 predicted. Manual inspection of top
(Olson/Norledge) Scoring by electrostatic, hydrophobic, van der scoring solutions. T01: Hpr S46 constrained near
Waals (buried surface area) interactions and kinase active site ⫹ manual check of kinase
shape complementarity. Clashes determined binding region. T02–T03: Ag epitopes from other
by overlap of the docked surfaces. structures. Penalization of Ag regions inaccessible
in virion. Manual filtering of symmetric solutions.
Universidade Nova De BiGGER (Chemera) Binary grids; scoring with geometric, contacts Some manual ranking. T07: TCR. truncated at D118
Lisboa counts, electrostatics and solvation, no clash
(Palma/Krippahl) checks.
Aberdeen U. UK (Ritchie) Hex Spherical polar Fourier correlation of shape and T01: search restricted to HPrK P-loop. T02–T07:
electrostatics, followed by soft rigid body OPLS search restricted to antibody and TCR
minimization. hypervariable loops. Visual inspection and
selection of orientations for T01–T03.
North Western U. US Northwestern DOCK Search for hot-spot correspondences between Only T01 predicted
(Shoichet) receptor and ligand to calculate orientations,
precalculation of ensemble of ligand
conformations, receptor held rigid, energy
evaluation using electrostatics and van der
Waals terms.
Imperial Coll. UK 3D-DOCK MULTIDOCK FFT; rescore: residue potentials; flexible Search restricted to expected interacting regions in all
(Sternberg) refinement: mean-field side-chain multicopy, cases (CDRs for targets T02–T06); manual
solution clustering selection from highest ranking solutions.
UCSD US (Ten Eyck) DOT FFT for shape complementarity and Poission– Ranking by shape complementarity (FADE). T02–
Boltzmann electrostatics. T07 screened for CDRs. Manual ranking for T01
and T02. Cluster analysis for T04–T07.
Kitasato U. JP TSCF Solvent cluster fitting; refinement by molecular No biochemical data, or bioinformatics used. Some
(Umeyama/Komatsu) mechanics and molecular dynamics using manual ranking.
AMBER force field.
SUNY/StonyBrook, US GRAMM FFT for shape complementarity; softer potential Only T01–T03 predicted. T01: filtered for HPr H15 or
(Tovchigrechko, for conformational changes; no clash check; D46 close to kinase active site. T021: filtered for for
Vakser) symmetry of multimeric receptors used to CDRs44 in contact ⫹ constraints on epitope
enhance sampling; docking of ligand to residues. T03: Filtered for CDRs45 in contact ⫹
overlapping fragments of receptor for speed. constraints on epitope
U. Autonoma Madrid, SP Neural network-based predictions of interactions Only T01 predicted
(Valencia) sites, using information on related sequences.
Boston U. US (Weng) ZDOCK FFT with scoring by pairwise shape Blocked non-CDR residues for T02–T06. T01: Ser 46
complementarity, solvation and electrostatics; of Hpr within 7 Å from P-loop. T02: distal half of
clustering. VP6. T07: locked residues according to homology;
manual selection from best solutions
Tel Aviv U., IL NIH, US PatchDock (T04–07) Geometric docking: matching of local shape T01: HPr Ser46 close to P on kinase. T02–T06: CDRs
(Wofson/Nussinov) BUDDA (T01–07) features and geometric hashing, fast geometric only. T04–T06: preference for high-sequence
PPD (T01) scoring and search, avoids exhaustive variability regions in mammalian amylases. T07:
orientation search. used interface from homologue.
1
Figure 6 of this reference.44
The left-most column lists the affiliation of the predictor group and the group principal investigator(s). The second column gives the acronym of
the docking software. Column 3 lists key features of the docking algorithm and the right-most column lists details on the various assumptions,
constraints, and additional information used in applying the different procedures to the CAPRI targets. The format of this table was inspired from
(http://www.bmm.icnet.uk/⬃smithgr/soft.html) with permission from G. Smith.
 BLIND PREDICTIONS OF PROTEIN–PROTEIN INTERACTIONS 55

in that complex. This latter quantity is informative be- TABLE II. Criteria for Ranking the CAPRI Predictions
cause a predicted complex in which a small number of
Rank fnat L_rms or I_rms
native contacts is embedded in a large number of non-
native ones is much less useful than a complex in which High ⱖ0.5 ⱕ 1.0 or ⱕ 1.0
the number of false-positive contacts is small. Medium ⱖ0.3 1.0 ⬍ x ⱕ 5.0 or 1.0 ⬍ x ⱕ 2.0
Acceptable ⱖ0.1 5.0 ⬍ x ⱕ 10.0 or 2.0 ⬍ x ⱕ4.0
When several quasi-symmetrical docking solutions were
Incorrect ⬍0.1
submitted, as was the case for targets T01–T03 in which
one of the components is a multisubunit protein, fnat was Column 1 lists the rank of the predictions, and columns 2 and 3 list the
parameter ranges defining the ranks. These parameters are the
evaluated for all the submitted solutions and the one with fraction of native contacts fnat defined as the number of native
the largest fnat value was used for further analysis. residue–residue contacts in the predicted complex divided by the
To evaluate the extent to which a prediction identifies number of native contacts in the target (see text). L-rms is the
the native interaction surfaces or “epitopes” on either or backbone root-mean-square displacement of the ligands in the pre-
dicted versus the target structures computed after the receptors of
both proteins, independently of its ability to produce the
these structures have been superimposed, and the corresponding
native contacts between them, we also computed the coordinate transformation has been applied to superimpose the two
fraction of native interface residues fIR. This fraction, complexes. The I-rms, is the root-mean-square displacement of the
computed for each of the interacting proteins, is defined as backbone of the interface residues only, in the predicted versus the
the number of native residues in the predicted interface target complexes (see text).
divided by the total number of native residues in the
corresponding target interface. Here, the interface resi-
dues were defined as those that lose accessible surface model fits the target structure in the interface region,
area when the two proteins associate. realizing well that such measure includes contributions
from rigid body motion as well as local conformational
Backbone RMSDs and Rigid Body Transformations changes. To that end, the interface residues in the target
were redefined as those having at least one atom within 10
To evaluate the overall geometric fit between the 3D Å of an atom on the other molecule. This is twice the
structures of the predicted and observed complexes, sev- threshold used in defining the residue–residue contacts
eral parameters were computed. One is the RMSD of the described above. The backbone of these residues were then
ligand (the smaller of the two proteins) in the predicted superimposed on their equivalents in the predicted com-
versus target complexes after the receptor (the larger of plex to compute the I_RMS.
the two proteins) was superimposed.41 This ligand RMSD,
denoted L_RMS, and the superpositions were both com- Atomic Clashes
puted on backbone atoms (N,C␣,C,O). To further character- Early during the analysis, it became obvious that sev-
ize the global fit between the predicted and observed eral prediction procedures produced complexes that fea-
ligand positions, we also computed the residual rigid body tured a very large number of atomic clashes due to
rotation angle ␪L and the residual translation vector of the appreciable interpenetration of the two molecules and that
geometric centre dL required to superimpose the ligand such interpenetration often artificially increased the frac-
molecules once the receptors have been superimposed. tion of native contacts fnat. Therefore, it was agreed to
Because some predictors chose to perform their docking impose a threshold on the allowed number of clashes for
calculations by using only part of the component molecules the predictions submitted for a given target. To that end,
(e.g., the hypervariable domains of the Fab fragments), the we computed the average and standard deviation of the
L_RMS values were computed for the molecular portions number of such clashes in all the predictions for each
that were common to all the predictors. In target T07, target and rejected from any further assessment predic-
where the two domains of the TCR receptor move relative tions where the number of clashes was 2 SD away from the
to one another on complex formation, the L_RMS was average. Clashes were defined as contacts between non-
computed on the backbone of the variable domain only. hydrogen atoms separated by ⬍3.0 Å.
When several quasi-symmetrical docking solutions were
possible or actually submitted, as was the case for targets Ranking the Predictions
T01–T03, the receptor subunit involved in the maximum As initially suggested by members of the CAPRI manage-
number of residue–residue contacts with the ligand was ment committee, a major criterion for ranking the pre-
successively superimposed onto the different receptor sub- dicted complexes was the fraction of native residue–
units of the target. The superposition yielding the best fit residue contacts (fnat). But because this quantity is sensitive
of the ligands was then used to computed the L_RMS and to the number of atomic clashes and depends on the chosen
other related parameters. distance cutoff parameter, a detailed evaluation was per-
The L_RMS is a global measure that depends on the size formed on all the predictions with fnat ⱖ 0.1, whereas those
of the ligand. Thus, it may not always provide a good with lower fnat values were not evaluated further because
picture of the fit at the interface, especially when the they were considered to represent random solutions.
ligand is large and oriented slightly differently in the The evaluated predictions were in turn grouped into
predicted model than in the target. Therefore, it was four categories on the basis of the three parameters fnat,
agreed to evaluate also the backbone RMSD of interface L_RMS, and I_RMS. These categories are highly accurate,
residues only (I_RMS) and use it as a measure of how the medium accuracy, acceptable, and incorrect, with predic-
56 R. MÉNDEZ ET AL.

TABLE III. CAPRI Prediction Results for Individual Targets

A. Target T01
Lactobacillus HPrK - B. subtilis HPr complex
Predictor groups 16
Evaluated predictions 69
High accuracy(***) 0
Medium accuracy(**) 0
Acceptable (*) 8
Incorrect 55
Predictions with clashes 6
Average no. of clashes (SD) 80.4 (84.3)
Model no.
(category) Predictors fnat fnon-nat fIR_L fIR_R Nclash L_rms (Å) I_rms (Å) ␪L (°) dL (Å)
5 (*) U Sheffield UK 0.3 0.9 0.5 0.6 215 12.0 3.1 49.4 9.5
(Gardiner)
1 (*) Weizmann Inst. IL 0.2 0.8 0.6 0.4 27 7.5 2.5 34.1 4.8
(Eisenstein)
1 (*) U. Autonoma Madrid SP 0.2 0.9 0.7 0.8 140 8.5 3.9 47.9 3.7
(Valencia)
1 (*) Boston U. US 0.2 0.9 0.7 0.6 103 9.4 4.0 56.6 2.4
(Camacho).
3 (*) Tau IL/NIH US 0.2 0.8 0.7 0.8 45 10.2 3.4 43.1 7.6
(Wolfson)
1 (*) Scripps US 0.1 0.9 0.6 0.6 76 10.0 3.3 44.9 7.6
(Olson)
3 (*) UCSD US 0.1 0.9 0.5 0.3 46 15.9 3.9 64.1 12.2
(Ten Eyck)
B. Target T02
Bovine Rotavirus VP6-FAB complex
Predictor groups 15
Evaluated predictions 70
High accuracy (***) 0
Medium accuracy(**) 1
Acceptable (*) 6
Incorrect 60
Predictions with clashes 3
Average no. of clashes (SD) 59 (46.6)
Model no.
(category) Predictors fnat fnon-nat fIR_L fIR_R Nclash L_rms (Å) I_rms (Å) ␪L (°) dL (Å)
3 (**) Boston U. US 1.0 0.3 1 1 86 2.3 1.2 4.8 2.1
(Weng)
3 (*) U. Sheffield UK 0.5 0.6 0.8 0.8 84 9.2 2.4 29.9 6.1
(Gardiner)
3 (*) SUNY/MUSC US 0.4 0.6 0.8 0.8 12 12.9 3.9 38.8 8.1
(Vakser)
1 (*) Weizmann IL 0.4 0.7 0.8 0.9 33 8.5 3.4 15.5 7.5
(Eisenstein).
1 (*) Imperial Coll. UK 0.4 0.4 0.6 0.8 8 11.8 3.7 36.7 7.0
(Sternberg)
4 (*) UCSD US 0.3 0.8 0.9 0.9 99 12.9 4.0 31.6 10.2
(Ten Eyck)
 BLIND PREDICTIONS OF PROTEIN–PROTEIN INTERACTIONS 57

TABLE III. (Continued)

C. Target T03
Influenza hemagglutinin-FAB complex
Predictor groups 13
Evaluated predictions 62
High accuracy(***) 0
Medium accuracy(**) 2
Acceptable (*) 0
Incorrect 58
Predictions with clashes 2
Average no. of clashes (SD) 138.3 (124.4)
Model no.
(category) Predictors fnat fnon-nat fIR_L fIR_R Nclash L_rms (Å) I_rms (Å) ␪L (°) dL (Å)
2 (**) Scripps US 0.7 0.3 0.5 0.7 22 4.6 1.9 11.7 3.4
(Abagyan)
4 (**) Aberdeen U. UK. 0.7 0.5 0.5 0.8 100 7.4 1.7 20.4 6.1
(Ritchie)

D. Targets T04,T05
Porcine ␣-amylase-CAbAM-D10 complex
Porcine ␣-amylase-CAbAM-07 complex
T04 T05
Predictor groups 13 13
Evaluated predictions 65 64
High accuracy (***) 0 0
Medium accuracy (**) 0 0
Acceptable (*) 0 0
Incorrect 60 60
Predictions with clashes 5 4
Average no. of clashes (SD) 34.2 (37.4) 37.9 (37.4)
E. Target T06
Porcine ␣-amylase-CAbAM-D9 complex
General summary
Predictor groups 13
Evaluated predictions 65
High accuracy (***) 4
Medium accuracy (**) 4
Acceptable (*) 0
Incorrect 54
Predictions with clashes 3
Average no. of clashes (SD) 34.5.7 (32.2)
Model no.
(category) Predictors fnat fnon-nat fIR_L fIR_R Nclash L_rms (Å) I_rms (Å) ␪L (°) dL (Å)
2 (***) Imperial Coll. UK 0.9 0.2 0.9 1.0 24 0.8 0.6 2.7 0.6
(Sternberg)
1 (***) Boston U. US 0.8 0.2 0.9 0.9 10 2.4 0.8 6.3 2.1
(Camacho)
5 (***) Aberdeen U. UK 0.8 0.1 0.9 0.9 20 2.2 0.8 7.8 1.5
(Ritchie)
5 (**) UCSD US 0.8 0.3 0.9 1.0 59 2.0 1.1 8.9 1.1
(Ten Eyck)
1 (**) U. Washington US 0.7 0.3 0.9 0.9 28 1.5 1.1 3.0 1.3
(Gray/Baker)
1 (***) Scripps US 0.7 0.4 0.9 0.9 4 1.3 0.9 5.6 0.7
(Abagyan)
5 (**) U. Lisbon PT 0.7 0.6 0.8 0.8 62 6.2 1.6 19.1 5.0
(Palma/Kripphal)
58 R. MÉNDEZ ET AL.

TABLE III. (Continued)

F. Target T07
TCR␤-SpeA complex
Predictor groups 14
Evaluated predictions 70
High accuracy (***) 5
Medium accuracy (**) 7
Acceptable (*) 8
Incorrect 47
Predictions with clashes 3
Average no. of clashes (SD) 33.8 (44.9)
Model no.
(category) Predictors fnat fnon-nat fIR_L fIR_R Nclash L_rms (Å) I_rms (Å) ␪L (°) dL (Å)
1 (**) Boston U. US 0.8 0.4 1.0 0.9 42 3.9 1.2 12.0 3.3
(Weng)
1 (***) Weizmann IL 0.8 0.2 1.0 0.8 12 2.1 0.9 7.2 1.3
(Eisenstein)
1 (***) Cancer Research UK 0.8 0.2 1.0 0.9 3 2.6 1.0 10.8 1.7
(Fitzjohn/Bates)
1 (***) Boston U. US 0.8 0.1 1.0 0.9 2 1.5 0.9 7.0 0.4
(Camacho)
1 (***) Tau IL/NIH US 0.7 0 1.0 0.9 5 1.3 0.6 3.4 0.6
(Wolfson)
1 (***) U. Washington US 0.6 0.2 0.9 0.9 5 2.2 0.7 7.1 1.5
(Gray/Baker)
5 (**) Scripps US 0.6 0.5 1.0 0.9 3 6.0 1.9 22.6 3.9
(Abagyan)
1 (*) U. Lisbon PT 0.5 0.4 0.9 0.8 13 7.5 2.3 31.6 4.8
(Palma/Krippahl)
3 (*) Imperial Coll. UK 0.3 0.8 0.6 0.8 15 8.5 3.3 38.1 4.5
(Sternberg)
Sections (a)–(f) of this table are devoted to the results for individual targets T01–T07. Each section is divided into two main parts. The top part
gives a general summary of the predictions, and the bottom part lists the key parameters of the best predictions ranked as acceptable or higher
submitted by each group.
The submitted predictions were divided into four categories as detailed in Table II. Predictions with a number of clashes exceeding a defined
threshold were not evaluated. Clashes were defined as those between two non-hydrogen atoms on each side of the interface whose distance was ⬍
3 Å. The threshold was taken as 2 SD, plus the average of the number of clashes in all the predictions submitted for a given target.
Detailed results for the best predictions for each participant, which were of acceptable quality or better (bottom), ranked as indicated in Table II.
Column 1 lists the model number (1–5) and the rank of the prediction (high accuracy (***), medium accuracy (**), and acceptable (*). Column 2
lists the participant groups in order of decreasing native contact fraction fnat (column 3). Column 3 lists the fraction of non-native contacts fnon-nat,
defined as the number of non-native contacts over the total number of contacts in the predicted complex. fIR_L/R is defined as the number of native
residues in the predicted interface over the total number of interface residues in the target, computed for both the R (receptor) or L (ligand)
molecules. Column 7 (Nclash) lists the number of atomic clashes in the predicted complex. Columns 8 and 9 list the RMSD values. L_RMS is the
backbone RMSD (Å) of the ligand molecules in the predicted versus the target complexes after the receptor moieties have been superimposed. The
I_RMS is the interface RMSD (Å) computed by superimposing only the backbone of the interface residues from the target complex onto their
counterparts in the predicted complex. The last two columns list the residual rigid body rotation (␪L) and translation (dL) of the ligand in the
predicted versus the target complexes after the corresponding receptor molecules have been superimposed. For further details on how the various
parameters were computed, see the text.

tions of the latter category being considered as equivalent A detailed description of all the parameters and all the
to random solutions. The parameter ranges used to define results obtained for the seven targets can be found on the
these categories are summarized in Table II. As expected, CAPRI web site (http://capri.ebi.ac.uk).
we could observe a correlation between the different
parameters, particularly between the native contact frac- Predictions for Individual Targets
tion and the I_RMS, although mainly for predictions Target T01: HPr-K/P-Hpr complex
reasonably close to the correct solution. But this correla-
tion tended to break down for poorer predictions. Table III(a) summarizes the prediction data for Target
01, the Hpr-K/P-Hpr complex. Sixteen groups submitted a
RESULTS AND DISCUSSION total of 69 nonredundant predictions for this target (predic-
In this section, the CAPRI evaluation results are pre- tions not related by quasi-symmetry). Of these, six had too
sented as follows. First, we describe the results obtained many atomic clashes (more than the threshold of 228
for individual targets. Second, an overview is presented of atom–atom contacts within 3 Å) and were not analyzed
the results across predictors and targets, respectively, and further. Of the remaining 63 predictions, only 8 were of
third, an attempt is made to relate the results of the acceptable quality (with fnat ⱖ 0.1, L_RMS ⱕ 10 Å and
different groups with the methodology used. I_RMS ⱕ 4 Å), and none were of medium or high accuracy.
 BLIND PREDICTIONS OF PROTEIN–PROTEIN INTERACTIONS 59

The best of the acceptable predictions obtained for this predictor reports in the present issue). An extreme case of
target by each group and their quality measures are listed this approach is the prediction by the Madrid (Valencia)
in Table III(a) (bottom). We see that the prediction with group, which involved no docking calculations at all but
the highest native contact ratio (fnat ⫽ 0.3), by U. Sheffield was entirely based on their neural network analysis of the
(Gardiner), has an appreciable number of clashes (215), residue conservation patterns, and used standard model-
and a large fraction of non-native contacts (fnon-nat ⫽ 0.9). ling to generate the structure of the complex.
It also features a quite large L_RMS (12 Å) of the Hpr
molecule whose overall rotation and translation are shifted Target T02: Rotavirus VP6-Fab Complex
by ⬃49° and 9.5 Å, respectively [last two column of Table Table III(b) summarizes the prediction data for this
III(a)]. Despite this, the I_RMS value is relatively low target. Fifteen groups submitted a total of 70 predictions.
(⬃3.1 Å), reflecting the fact that it measures the fit of the Of those, six were of acceptable quality, one was of medium
predicted versus target backbones for interface residues accuracy, and none of high accuracy. Three predictions
only and is not affected by parts of the molecule far way were rejected from the evaluation because of too many
from the interface. clashes.
One of the best predictions listed in Table III(a) is that The best prediction for this target, by the Boston U.
by the Weizmann (Eisenstein) group. However, this predic- (Weng) group [bottom part of Table III(b)], is of medium
tion has a somewhat lower native contact fraction (fnat ⫽ accuracy. It retrieves all the native contacts (fnat ⫽ 1),
0.2) but also a slightly lower, although still too high, together with about 30% of non-native contacts. The
fraction of non-native (fnon-nat ⫽ 0.8). It has furthermore L_RMS and I_RMS for this predictions are 2.33 Å and 1.16
significantly lower RMSDs (L_RMS ⫽ 7.5 Å, I_RMS ⫽ 2.45 Å, respectively, indicating a rather good fit of the predicted
Å), the deviation of the rigid body rotation and translation and target ligand molecules, as well as a close to excellent
parameters of the Hpr molecule are smaller (34° and 4.8 fit of the backbone of the interface residues. The rigid body
Å), and the number of clashes is low (27). Except for the rotation and translation of the ligand are also correspond-
poorer prediction, at the bottom of the list, the remaining ingly small (4.8° and 2.05 Å). Only the number of clashes is
ones listed in Table III(a) are of roughly comparable a little high (87).
quality, although several have a notable number of clashes. All the other predictions listed in Table III(b) are of
A pictorial overview of all the predictions provided for acceptable quality only. They have fnat values between 0.3
this target is presented in Figure 1. This figure shows the and 05, I_RMS values between 2.4 and 4 Å and L_RMS
positions of the ligand HPr molecules and their geometric values in the range of 8.5–12.9 Å.
centers relative to the receptor surface in the target (Fig. In all the predictions for this target, the requirement to
1), and the same centers in all the predicted complexes involve the CDRs of the Fab molecule in the interaction
after the receptor backbones in these complexes have been was used to limit the search space or to filter solutions
superimposed onto their counterpart in the target, as (Table I and predictor reports in this issue). Several of the
described above [Fig. 1(b)]. For the acceptable predictions, predictors also restricted the search to regions of the
the geometric centers of the ligand cluster more clearly receptor that were more likely to be accessible to antibod-
around the observed positions, whereas in incorrect predic- ies on the virus envelope. However, these restrictions were
tions (fnat ⱕ 0.1) they sample a much wider range of obviously not sufficient to direct most of the prediction
positions, corresponding to random solutions. Some of the procedures to correct solution. Therefore, the single me-
depicted solutions lie near the three-fold axis of the kinase dium accuracy prediction for this complex should be
trimer or at the interface between the top and bottom viewed as quite an achievement.
trimer layers. Figure 2 presents a pictorial overview of all the predic-
Thus, we see that even the best predictions for this tions for this target, which illustrates well the differences
target are of relatively poor quality, as judged by most of in the positions of the geometric centres of the correct and
the parameters listed in Table III(a). The fact that fnat is incorrect solutions, and particularly the close fit of the
between ⬃0.1 and 0.3 in these predictions and that the geometric centres of the single medium accuracy predic-
false-positive contacts represent as much as 80% of the tion for this target.
predicted ones in the best prediction, suggests that the
models would not be too useful for any detailed analysis of Target T03: Influenza Hemagglutinin-Fab Complex
the interactions. However, considering the large size of the Table III(c) summarizes the prediction data for this
receptor and the appreciable difference between the bound target. Thirteen predictor groups submitted a total of 62
and unbound kinase conformations, the fact that some nonredundant predictions. Of those, two were eliminated
solutions approach, albeit imperfectly, the correct mode of because of clashes, two were ranked medium accuracy,
association should be viewed as very encouraging. On the and the rest as incorrect. Key parameters of the two
other hand, it is important to mention that this encourag- correct predictions are listed in bottom part of Table III(c).
ing, although limited, success is primarily the fruit of The best prediction, by the Scripps (Abagyan) group,
using biochemical data (on the residues expected to be in identifies 70% of the native contacts, and the fraction of
contact) or clues from patches of conserved residues in non-native contacts in the predicted structure is rather
multiple alignments of related kinases and Hpr proteins to low (30%). The RMSD values (L_RMS ⫽ 4.6 Å, I_RMS ⫽
guide or filter the docking predictions (Table I and other 1.9 Å) and the rigid body shifts (11.7°, 3.4 Å) are
60 R. MÉNDEZ ET AL.

Figure 1.

Figure 2.
 BLIND PREDICTIONS OF PROTEIN–PROTEIN INTERACTIONS 61

relatively small too, with only 22 clashes. The second involving only the CDR regions or filtered out solutions
acceptable prediction has the same fraction of native that did not involve them in a postprocessing step, thereby
contacts, but the fraction non-native contacts is higher effectively eliminating any complexes with alternative
(50%), and so are the RMSD values and the number of interaction modes. It is of interest that a handful of
clashes. It is rather reassuring to see that both predic- submitted solutions did involve the CDR regions of the
tions are of higher quality than the blind predictions VHH domain and the amylase active site and thus pro-
submitted for a similar target in a previous contest 6 vided acceptable predictions for target T06, which does in
years earlier.30 fact involve such interaction (see below). But these solu-
Two other groups submitted one prediction each that tions were not assessed here. In contrast to the previous
qualified as acceptable, but these predictions were not targets, these negative results illustrate how restrictions
assessed here because they were not among the top five and filters applied on the basis of prior knowledge can just
solutions submitted by these groups. as effectively confine the search to incorrect solutions.
Here too, most predictors limited the search regions to A much better performance was observed for Target
the loops of the hypervariable regions of the antibody and T06, which features the expected mode of interactions with
to the accessible portions of the antigen. But as for target the VHH domain. Thirteen groups provided a total of 65
2, the correct solutions certainly cannot be attributed predictions for this target [Table III(e)] and three were
solely to these restrictions, especially because virtually all rejected as having too many clashes. As many as four
the predictors used them. predictions were ranked as highly accurate, four as me-
A pictorial presentation of the target and a summary of dium accuracy models, and the remainder as incorrect
the predictions for this target is presented in Figure 3. It solutions.
shows that the scatter of the ligand geometric centers is The best prediction of acceptable quality or better pro-
similar to that in target T02. vided for this target by each group, together with their
quality measures, are listed in Table III(e). The most
Target T04-T06: Complexes Between Pancreatic ␣-
accurate prediction of the lot, and for that matter of the
Amylase and Camelid Antibodies
entire challenge, was that by the Imperial Coll. (Sternberg)
Respectively, 65 and 64 predictions were submitted for group. This highly accurate prediction retrieves 90% of the
targets T04 and T05. All of them were qualified as native contacts, with a mere 20% of false positives, has
incorrect, with only three predictions of target T05 having RMSD values below 1 Å (L_RMS ⫽ 0.8 Å, and I_RMS ⫽ 0.6
0 ⬍ fnat ⱕ 0.1, and the remaining ones all had fnat ⫽ 0 Å) and only 24 clashes. Three other predictions in Table
[Table III(d)]. III(e), by Boston U. (Camacho), Aberdeen U. (Richie), and
This poor result is primarily due to the fact that the Scripps (Abagyan), are also of high accuracy. They identify
intermolecular contacts in the crystal structure involve between 70 and 80% of the native contacts and have only
the framework regions of the camelid VHH domain and 10 – 40% false-positive contacts. Although their L_RMS’s
not mainly the CDR region, as expected. Indeed, most range between 1.3 and 2.4 Å, their I_RMSs are all ⬍1 Å.
predictors either restricted the search space to solutions Therefore, all are rather accurate predictions by our
standards.
The remaining three predictions listed in Table III(e)
are of medium accuracy and hence also rather good. Thus,
Fig. 1. Pictorial overview of the prediction results for target T01, the
Hpr-K/P-Hpr complex. a: A top view of the target complex showing the looks like submitted solutions for this target were either
Hpr-K/P trimer (receptor) in ribbon representation with each subunit correct or random solutions, with few solutions in between.
represented in a different colour. The three bound Hpr ligand molecules A possible reason for this two-tier performance might be
are represented by their molecular surface envelope, and the red spheres
on each molecule depict the geometric centre of the ligands. b: Positions that in this case the receptor is a single protein whose
of the geometric center of the ligands relative to the kinase trimer in surface is less complex and features fewer major grooves
submitted predictions. To produce this picture, the receptor trimers in than the surfaces of the multiprotein receptors of targets
each predicted complex were superimposed onto their counterpart in the
target and the corresponding rigid body transformation was applied to the T01–T03. In addition, some predictors may have intro-
entire complex (see text). The turquoise spheres correspond to accept- duced additional constraints on the basis of evidence from
able solutions, and the gold spheres to incorrect solutions, following the
ranking criteria in Table II. The red spheres represent the relative biochemical studies34 that, unlike the two monoclonal
positions of the geometric centres of the ligands in the target structure. camelid antibodies of targets T04 and T05, this particular
This figure and Figures 2–5 were generated by using the Insight one inhibits enzymatic activity and is hence likely to bind
software.47
Fig. 2. Pictorial overview of the prediction results for target T02, the to the enzyme active site.
rotavirus VP6-Fab complex. a: The rotavirus receptor structure is in Figure 4 highlights the ligand geometric centre positions
ribbon representation with each subunit in the trimer represented in a in target T06 [Fig. 4(a)] and provides a pictorial summary
different color (bottom), and one bound Fab molecule represented by its
molecular surface envelope (top). The red sphere indicates the geometric of the predictions [Fig. 4(b)], which illustrate rather well
center of the variable domain of the antibody used in analyzing the the good fit of the geometric centres.
correspondence between the predicted and observed ligand positions. b:
Positions of the geometric centers of the ligands relative to the VP6 trimer Target T07: TCR␤-SpeA Complex
in predicted complexes ranked as being of medium accuracy (blue
spheres), acceptable (turquoise spheres) and incorrect (gold spheres). Fourteen groups submitted 70 predictions for this tar-
The relative position of the geometric center of the ligand in the target is
shown as a red sphere. This picture was produced as described in the get. Five were of high accuracy, seven of medium accuracy,
legend of Figure 1(b). eight were qualified as acceptable, and three were dis-
62 R. MÉNDEZ ET AL.

Figure 3.

Figure 4.
 BLIND PREDICTIONS OF PROTEIN–PROTEIN INTERACTIONS 63

Fig. 5. Pictorial overview of the prediction results for target T07, the TCR␤-SpeA complex. a: The target complex, depicting ribbon drawings of the
receptor (SpaA molecules) in green, and the TCR ␤ chain ligand in purple. The geometric center of the ligand portion used for the analysis (TCR␤
variable domain) is depicted as a red sphere. b: Positions of the geometric center of the ligands relative to the superantigen in predictions ranked as
highly accurate (green spheres), medium accuracy (blue spheres), acceptable (turquoise spheres), and incorrect (gold spheres). For all other details,
see legend of Figure 1(b).

carded for having too many clashes [Table III(f)]. It is of Table III(f) (bottom) lists the details for the best predic-
interest for this target more than any other, several of the tion ranked acceptable or higher from each group. The five
groups submitted more than one correct prediction, and for highly accurate predictions identify between 60 and 80% of
two groups Tau (Wolfson) and Boston U. (Weng), all the the native contacts and between 0 and 20% of false-
five submitted predictions were of acceptable quality or positive contacts. Their L_RMSs are between 1.3 and 2.6 Å
better. and I_RMSs are all 1 Å or less. The two medium accuracy
predictions by the Boton U. (Weng) and Scripps (Abagyan)
groups have native contacts fraction of 80% and 60%,
Fig. 3. Pictorial overview of the prediction results for target T03, the
respectively, but a larger proportion of false-positive con-
influenza Hemagglutinin-Fab complex. a: The influenza hemagglutinin tacts (40 –50%) and higher L_RMS values (3.9 – 6.0 Å). The
structure is in ribbon representation, with each subunit in the trimer Scripps prediction is a borderline case, which is probably
represented in a different color (bottom) and the bound Fab molecules
represented by their molecular surface envelopes (top). The red spheres better ranked as acceptable, because it resembles more the
indicate the geometric center of the variable domains of the antibody used last two acceptable predictions listed in Table III(f) than
in analyzing the correspondence between the predicted and observed
ligand positions. b: Positions of the geometric center of the ligands the better models.
relative to the hemagglutinin trimer in predictions ranked as being of The prediction performance for this target (20 models of
medium accuracy (blue spheres) and incorrect (gold spheres). For other
details see legend of Figure 1(b).
acceptable or higher quality out of a total of 70 models) is
Fig. 4. Pictorial overview of the prediction results for target T06, the thus relatively high and clearly higher than for targets
porcine ␣-amylase-CAbAM D9 complex. a: The target complex, depicted T01–T06. Inspection of the prediction methods summary
as ribbon drawings of the porcine ␣-amylase receptor in green, and the
CabAM-D9 VHH ligand in purple. The geometric center of the ligand is in Table I reveals that all the groups that produced at least
depicted as a red sphere. b: Positions of the geometric center of the one high or medium accuracy prediction for target T07
ligands relative to the enzyme in predictions ranked as being highly used the information on the known 3D structure of the
accurate (green spheres), medium accuracy (blue spheres), acceptable
(turquoise spheres), and incorrect (gold spheres). For all other details see homologue complex to guide, or filter, their solutions. The
legend of Figure 1(b). groups that did not use this information produced lower
64 R. MÉNDEZ ET AL.

TABLE IV. Summary of Docking Predictions

Predictor group T1 T02 T03 T04 T05 T06 T07 Predictor summary
Scripps US 0 0 ** 0 0 *** ** 3/2**/1***
(Abagyan)
Boston U. US * 0 0 0 0 *** *** 3/2***
(Camacho)
Weizmann Inst. IL * * 0 0 0 0 *** 3/1***
(Eisenstein)
Imperial Coll. UK 0 * 0 0 0 *** * 3/1***
(Sternberg)
UCSD, US * * 0 0 0 ** 0 3/1**
(Ten-Eyck)
Sheffield U. UK * * — — — — — 2
(Gardiner)
U. Washington US 0 0 0 0 0 ** *** 2/1**/1***
(Gray/Baker)
U. Lisbon,Pt — 0 — 0 0 ** * 2/1**
(Palma/Krippahl)
Aberdeen U. UK. UK 0 0 ** 0 0 *** 0 2/1**/1***
(Ritchie)
Boston U. US 0 ** 0 0 0 0 ** 2/2**
(Weng)
TAU IL /NIH US * 0 0 0 0 0 *** 2/1***
(Wolfson/Nussinov)
Cancer Research UK — — — 0 0 0 *** 1/1***
(Fitzjohn/Bates)
Scripps US * 0 0 — — — — 1
(Olson)
U. Autonoma Madrid SP * — — — — — — 1
(Valencia)
SUNY/MUSC US 0 * 0 — — — — 1
(Vakser)
Target summary 7 6/1** 2/2** 0 0 7/3**/4*** 9/2**/5***
This table summarizes the results obtained by all the groups that submitted one or more predictions of acceptable quality or better for at least one
target. Column 1 lists the group’s affiliation and the last name of the principle investigator. The next seven columns list the results obtained for
each of the seven targets. The right-most column summarizes the results per predictor group, and the bottom row summarizes the results per
target. 0, none of the submitted predictions was of acceptable quality;—,no predictions were submitted; *, at least one of the submitted predictions
was in the acceptable range; **, at least one of the submitted predictions was of medium accuracy; and ***, at least one prediction was of high
accuracy. See Table II for the definition of the parameters range used to rank the predictions. The summary entries list the total number of
acceptable predictions, followed by the number of predictions of medium and high accuracy denoted by a ** and ***, respectively.

quality or incorrect predictions. The pictorial summary of one acceptable, or better, prediction for targets T06 and
the predictions for this target is given in Figure 5. T07. One provides a medium accuracy prediction for target
T03, whereas four have one acceptable prediction for
either target T01 or T02, or both.
Overview of the Results Across Predictor Groups
A further six groups submitted acceptable predictions
and Targets
for only two targets, and for half of these, the best
Table IV summarizes the results obtained for all seven prediction is of medium or high accuracy. The Sheffield
targets by all the groups that submitted at least one (Gardiner) group submitted predictions for only two tar-
prediction ranking acceptable or better. We see that 14 of gets (T01 and T02) and those contained acceptable solu-
the 19 groups taking part in this first CAPRI experiment tions for both.
have an entry in this table. Inspection of Table IV reveals The last four entries in Table IV are for groups that
moreover that five of the seven targets were correctly submitted predictions for only a small subset of the
predicted, albeit, with varying accuracy, by at least one targets. Thus, it is not too surprising that they produced
group. only one acceptable or better prediction. The Madrid
The best any group has done is to provide at least one (Valencia) group predicted only target T01 because it
acceptable prediction for three of the seven targets avail- seemed to be the only case amenable to their neural
able for this experiment, although obviously several of the network approach. The Scripps (Olson) and SUNY (Vak-
groups listed in Table IV did not submit predictions for ser) groups did not submit predictions to the second round
every target. The first five groups listed in Table IV of CAPRI (targets T04 –T07), whereas the Cancer Re-
achieve this performance. Three of these provide at least search (Bates) group joined only the second CAPRI round
 BLIND PREDICTIONS OF PROTEIN–PROTEIN INTERACTIONS 65

and hence provided no predictions of the first three tar- the descriptions of the docking procedures in Table I with
gets. the results summary in Table IV. We see indeed that the
It can also be seen that no highly accurate predictions most successful procedures, those that produced at least
were obtained for the first three targets, which represent three predictions of acceptable or higher quality, are based
the more difficult prediction problems either due to the on very different search algorithms (FFT, rigid body
large size of the molecules, or to conformational differences mechanics molecular mechanics) and scoring functions
between the bound and unbound components, or both. On (geometric only, different empirical, and database-derived
the other hand, highly accurate predictions were achieved force fields). Because several other procedures, which
by several groups for targets T06 and T07. produced less successful results, used similar algorithms
The excellent results obtained for Target T07 were or scoring function in combination with biochemical knowl-
clearly the consequence of using structural information on edge, it is tempting to conclude that a successful achieve-
a related complex. But those for target T06 made use only ment depends on the manner in which this knowledge is
of biochemical information on the regions likely to interact integrated into the calculation procedure. Thus, more
(VHH CDRs with the enzyme active site) and are therefore work in this area is clearly a direction for future research.
more directly attributable to the docking procedures them- A legitimate question to address at this stage is under
selves. The use of this type of information, as well as data which circumstances and for what purpose can current
on sequence conservation, were most probably key to docking procedures be useful?
having as many as seven groups producing acceptable Having found that accurate predictions can be obtained
predictions for Target T01 despite the important conforma- for only a subset of problems (molecules are small, limited
tional differences between the bound and unbound kinase conformational changes, biochemical information avail-
molecules. able, etc.), we can conclude that, at least for such problems,
Finally, a comment is warranted on the two groups current docking procedures are capable of producing mod-
Columbia (Norel) and Kitasato U. (Umeyama) who submit- els accurate enough for guiding drug design or for rational
ted predictions for all seven targets but produced accept- mutagenesis studies. Given that usually these models
able predictions for none. The main feature common to accurately predict the relative ligand-receptor position
their approaches seemed to be that they made little or no and orientation as well, they should also be useful in
use of biochemical information. Otherwise, their docking structural studies for deriving phase information by mo-
procedures are very different, with the Columbia team lecular replacement42 or for the interpretation of low-
using a more classical FFT procedure, whereas the Kita- resolution electron density maps obtained by electron
sato team used a new hitherto untested algorithm (Table microscopy.40
I). Nothing can be said about the two remaining groups Medium-accuracy predictions seem to be reachable for a
with no acceptable predictions, Aberdeen (Mustard), and wider range of problems, including those with quite large
Northwestern (Shoichet), because both submitted predic- receptor molecules. But here too, biochemical information
tions for a single target, T01 and T07, respectively. and prior knowledge are needed to guide the calculations.
However, the resulting models often contain between 30
Current Status of Docking Procedure and Their
and 60% of non-native contacts, and although the interface
Potential Applications
RMSDs are within 2 Å, detailed side-chain modeling might
This first CAPRI experiment with 465 predictions for not be straightforward. Therefore, such models are prob-
seven targets, obtained by using a wide variety of methods, ably not accurate enough for guiding rational drug design
provides a much better overview of the performance of or structural studies but should be useful for exploratory
current docking procedures than hitherto available. The mutagenesis and functional analyses.
good quality predictions obtained here for a number of To enable such applications, however, the correct model
difficult targets clearly shows that progress has been should be among the top ranking solutions produced by the
achieved since docking procedures have last been assessed docking calculations, a condition largely fulfilled by the
about 6 years ago.28 –30 But the performance of current acceptable solutions analyzed here, because those were
procedures taken globally is by no means reliable enough among the five top predictions of each group examined in
to allow their use as a routine tool. For many real-world this evaluation. Moreover, of the 24 high- and medium-
problems, as in the case of targets T01–T06, only around accuracy predictions produced for the seven targets, a
10% of the submitted predictions are of acceptable quality sizeable fraction (60%) corresponds to the top ranking
or better. Only when detailed structural information is solutions (models 1 or 2). However, one should recall that
available on a related complex in which the interaction many predictors reranked their predictions manually
mode is conserved, as for Target 07, do we see a higher (Table I).
success rate of 30% correct predictions. Finally, many of the lower accuracy predictions, some-
Prior knowledge and information on the regions likely to what generously qualified as acceptable in this evaluation,
interact have been used whenever possible on all targets might also be very helpful. We find that many such
by most predictors (Table I). The results were positive in predictions, as well as some of the incorrect predictions,
most cases, and predictors who did not consider them often identify at least 50% of the native residues that
clearly lost out. A further indication that using such participate in the interaction on the receptor, the ligand or
information was important can be obtained by comparing both [fIR_L and fIR_R in Table III(a–f)]. Hence, many of the
66 R. MÉNDEZ ET AL.

predictions may be helpful in delineating regions on the ship. We also thank all the participants and the manage-
protein surface that are likely to interact. On the other ment team of this first CAPRI experiment for helpful
hand, it is not clear from the present analysis to what discussions and valuable input and cooperation. Last but
extent current docking procedures can be used for predict- not least, we express gratitude to all the crystallographers
ing if two proteins that are not known to interact before- that provided their X-ray structures before publication.
hand are likely to do so or not. This requires a systematic
REFERENCES
comparison of scores of good solutions with those of
random ones, something that was not performed here, and 1. Kleanthous C. Protein-protein recognition, Oxford University
Press; 2000.
undertaken by a few groups only (Eisenstein et al. this
2. Janin J, Wodak SJ. The structural basis of macromolecular
issue). recognition. Adv Protein Chem 2002;61:9 –73.
3. Jones S, Thornton JM. Principles of protein-protein interactions.
Prospects for Improving Docking Methods Proc Natl Acad Sci USA 1996;93:13–20.
4. Marcotte EM, Pellegrini M, Thompson MJ, Yeates TO, Eisenberg
The very wide panorama of methods used in this first D. A combined algorithm for genome-wide prediction of protein
CAPRI experiment offers unprecedented opportunity for function. Nature 1999;402:83– 86.
5. Xenarios I, Rice DW, Salwinski L, Baron MK, Marcotte EM,
some useful insights into the directions for future progress. Eisenberg D. DIP: the database of interacting proteins. Nucleic
Not too surprising is the recognition that the ability of Acids Res 2000;28:289 –291.
docking methods to single out the correct modes of protein– 6. Enright AJ, Iliopoulos I, Kyrpides NC, Ouzounis CA. Protein
interaction maps for complete genomes based on gene fusion
protein interactions from a large number of incorrect ones events. Nature 1999;402:86 –90.
is still limited. However, appreciable efforts in this direc- 7. Bader GD, Donaldson I, Wolting C, Ouellette BF, Pawson, T,
tion are being made already, as judged by the increased Hogue CW. BIND—the biomolecular interaction network data-
sophistication of the scoring functions used in several of base. Nucleic Acids Res 2001;29:242–245.
8. Park J, Lappe M, Teichmann SA. Mapping protein family interac-
the more recent procedures described in Table I. Further tions: intramolecular and intermolecular protein family interac-
progress should be expected as our capacity to judiciously tion repertoires in the PDB and yeast. J Mol Biol 2001;307:929 –
approximate the physical contributions to the free energy 938.
9. Uetz P, Giot L, Cagney G, Mansfield TA, Judson RS, Knight JR,
of these interactions improves. Lockshon D, Narayan V, Srinivasan M, Pochart P. A comprehen-
Another key component of docking procedures is the sive analysis of protein-protein interactions in Saccharomyces
algorithms for sampling rigid body and conformational cerevisiae. Nature 2000;403:623– 627.
10. Ito T, Chiba T, Ozawa R, Yoshida M, Hattori M, Sakaki Y. A
degrees of freedom. Some of these, in particular the comprehensive two-hybrid analysis to explore the yeast protein
spherical polar Fourier correlation or geometric hashing interactome. Proc Natl Acad Sci USA 2001;98:4569 – 4574.
algorithms, are quite efficient. But often the procedures 11. Hazbun TR, Fields S. Networking proteins in yeast. Proc Natl
that implement them are not the same as those with the Acad Sci USA 2001;98:4277– 4278.
12. Rain JC, Selig L, De Reuse H, Battaglia V, Reverdy C, Simon S,
most advanced scoring functions. Therefore, notable Lenzen G, Petel F, Wojcik J, Schachter V. The protein-protein
progress could result in the near future from combining interaction map of Helicobacter pylori. Nature 2001;409:211–215.
the best of both worlds. 13. Wodak SJ, Janin J. Computer analysis of protein-protein interac-
tion. J Mol Biol 1978;124:323–342.
Handling conformational flexibility remains nonethe- 14. Janin J, Wodak SJ. Reaction pathway for the quaternary struc-
less a major challenge. However, several completely novel ture change in hemoglobin. Biopolymers 1985;24:509 –526.
procedures evaluated in this first CAPRI experiment were 15. Halperin I, Ma B, Wolfson H, Nussinov R. Principles of docking:
capable of handling some conformational flexibility and an overview of search algorithms and a guide to scoring functions.
Proteins 2002;47:409 – 43.
simulating more physical docking trajectories. Some of 16. Jiang F, Kim SH. “Soft docking”: matching of molecular surface
these seemed quite promising because they performed no cubes. J Mol Biol 1991;219:79 –102.
worse than the more classical approaches despite being 17. Cherfils J, Duquerroy S, Janin J. Protein-protein recognition
analyzed by docking simulation. Proteins 1991;11:271–280.
still in an early stage of development. 18. Shoichet BK, Kuntz ID. Protein docking and complementarity. J
Thus, there appears to be not only a lot of room for Mol Biol 1991;221:327–346.
improvements in docking algorithms but also concrete 19. Katchalski-Katzir E, Shariv I, Eisenstein M, Friesem AA, Aflalo
C, Vakser IA. Molecular surface recognition: determination of
perspectives in achieving them. Large-scale blind predic- geometric fit between proteins and their ligands by correlation
tions and their objective assessment, as done here, are techniques. Proc Natl Acad Sci USA 1992;89:2195–2199.
clearly key in monitoring progress in this field. Therefore 20. Norel R, Fischer D, Wolfson HJ, Nussinov R. Molecular surface
it is of paramount importance that other CAPRI experi- recognition by a computer vision-based technique. Protein Eng
1994;7:39 – 46.
ments should follow this one to create a similar momen- 21. Totrov M, Abagyan R. Detailed ab initio prediction of lysozyme-
tum as that produced by CASP for protein structure antibody complex with 1.6 A accuracy. Nat Struct Biol 1994;1:259 –
prediction methods. 263.
22. Vakser IA. Protein docking for low-resolution structures. Protein
Eng 1995;8:371–377.
ACKNOWLEDGMENTS 23. Meyer M, Wilson P, Schomburg D. Hydrogen bonding and molecu-
lar surface shape complementarity as a basis for protein docking.
We acknowledge the Marie-Curie Host Training grant J Mol Biol 1996;264:199 –210.
(contract num, QLK3-1999-51297), to Raúl Méndez for 24. Gabb HA, Jackson RM, Stemberg MJ. Modelling protein docking
support. Thanks go to Jean Richelle for help with com- using shape complementarity, electrostatics and biochemical infor-
puter systems and to the rest of the SCMBB (Service de mation. J Mol Biol 1997;272:106 –120.
25. Ackermann F, Hermann G, Posch S, Sagerer G. Estimation and
Conformation de Macromolecules Biologiques, et Bioinfor- filtering of potential protein-protein docking positions. Bioinformat-
matique) group members for their kind support and friend- ics 1998;14:196 –205.
 BLIND PREDICTIONS OF PROTEIN–PROTEIN INTERACTIONS 67

26. Camacho CJ, Weng Z, Vajda S, DeLisi C. Free energy landscapes MC, Cohen J, Kohli E, Pothier P, Hewat E. Antibody Inhibition of
of encounter complexes in protein-protein association. Biophys J the transcriptase activity of the rotavirus DLP: a structural view.
1999;76:1166 –1178. J Mol Biol 2001;307:161–172.
27. Ritchie DW, Kemp GJ. Protein docking using spherical polar 38. Wang H. Grid-search molecular accessible surface algorithm for
Fourier correlations. Proteins 2000;39:178 –194. solving the protein docking problem. J Comp Chem 1991;12:746 –
28. Strynadka NC, Eisenstein M, Katchalski-Kazir E, Shoichet BK, 750.
Kuntz ID, Abagyan R, Totrov M, Janin J. Zimmerman F. Molecu- 39. Delhaise P, Bardiaux M, De Maeyer M, Prevost M, Van Belle D,
lar docking programs successfully predict the binding of a beta- Donneux J, Lasters I, Van Cutsem E, Alard P, Wodak SJ. The
lactamase inhibitory protein to TEM-1 beta-lactamase. Nat Struct BRUGEL package: toward computer-aided design of macromol-
Biol 1996;3:233–239. ecules. J Mol Graphics 1988;6:219.
29. Dixon JS. Evaluation of the CASP2 docking session. Proteins 40. Wriggers W, Milligan RA, McGammon JA. Situs: a package for
1997;Suppl 1:198 –204. docking crystal structures into low-resolution maps from electron
30. Vakser IA. Evaluation of GRAMM low ressolution docking meth- microscopy. J Struct Biol 1999;125:185–195.
odology on the hemaglutinin-antibody complex. Proteins 1997; 41. McLachlan AD. Gene duplications in the sructural evolution of
Suppl 1:226 –230. chymotrypsin. J Mol Biol 1979;128:49 –79.
31. Moult J. Predicting protein three-dimensional structure. Curr 42. Navaza J. Implantation of molecular replacement in AMoRe. Acta
Opin Biotechnol 1999;10:583–588. Crystallogr D. Biol Crystallogr 2001;57:1367–1372.
32. Fieulaine S, Morera S, Poncet S, Mijakovic I, Galinier A, Janin J, 43. de Groot BL, van Aalten DMF, Scheek RM, Amadei A, Vriend G,
Deutscher J, Nessler S. X-ray structure of a bifunctional protein Berendsem HJC. Prediction of protein conformational freedom
kinase in complex with its protein substrate HPr. Proc Natl Acad from distance constraints. Proteins 1997;29:241–251.
Sci USA 2002;99:13437–13441. 44. Thouvenin E, Schoehn G, Rey F, Petitpas I, Mathieu M, Vaney
33. Barbey-Martin C, Gigant B, Bizebard T, Calder LJ, Wharton SA, MC, Cohen J, Kohli E, Pothier P, Hewat E. Antibody inhibition of
Skehel JJ, Knossow M. An antibody that prevents the hemaggluti- the transcriptase activity of the rotavirus DLP: a structural view.
nin low pH fusogenic transition. Virology 2002 Mar 1;294:70 –74. J Mol Biol 2001;307:161–172.
34. Desmyter A, Spinelli S, Payan F, Lauwereys M, Wyns L, Muylder- 45. Gigant B, Barbey-Martin C, Bizebard T, Fleury D, Daniels R,
mans S, Cambillau C. Three camelid VHH domains in complex Skehel JJ, Knossow. A neutralizing antibody Fab-influenza haem-
with porcine pancreatic alpha-amylase. Inhibition and versatility agglutinin complex with an unprecedented 2:1 stoichiometry:
of binding topology. J Biol Chem 2002;277:23645–23650. characterization and crystallization. Acta Crystallogr D 2000;56:
35. Sundberg EJ, Li H, Llera AS, McCormick JK, Tormo J, Schlievert 1067–1069.
PM, Karjalainen K, Mariuzza RA. Structures of two streptococcal 46. Duncan B, Olson A. Applications of evolutionary programming for
superantigens bound to TCR beta chains reveal/diversity in the the prediction of protein-protein interactions. Evolutionary pro-
architecture of T cell signaling. Structure (Camb) 2002;10:687– gramming V. Proceedings of the 5th annual conference on evolu-
699. tionary programming. Boston, MA: MIT Press; 1996.
36. Lo Conte L, Chothia C, Janin J. The atomic structure of protein- 47. Dayringer, H.E., Tramontano A., Sprang, S.R., Fletterick, R.J.,
protein recognition sites. J Mol Biol 1999;285:2177–2198. Interactive program for visualization and modeling of proteins,
37. Thouvenin E, Schoehn G, Rey FA, Petitpas I, Mathieu M, Vaney nucleic-acids and small molecules. J Mol Graph 1986;4:82.