Braille for Pugilists

Introduction.

What is a good model ?

• chemical: bond lengths, bond angles, etc. have reasonable values, chiral carbons have the correct hand, aromatic rings, peptide bonds and conjugated systems are essentially flat.
  
• physical: there are no bad contacts, the packing of the molecules in the cell is good, and related atoms (NCS-related, covalently linked, hydrogen-bonded, involved in a salt link) have similar temperature factors.
  
• crystallographic: the model adequately explains the experimental data, with as little over-fitting as possible.
  
• protein structural science: the protein has some secondary structure, the Ramachandran plot has few if any outliers, peptide oxygens have the correct orientation, the large majority of side chains have a rotamer conformation, salt links and hydrogen bonds make sense, there are no buried charges (except for the odd aspartate), waters and ions are properly coordinated, most if not all peptide bonds are trans, etc.
• statistical: the model constitutes the best hypothesis to explain the data, with the minimum degree of over-fitting.
  
• biological: the disulfide pairings make sense, and biochemical observations with respect to the effect, of for instance, mutations on the folding or activity can be explained.

Accuracy and precision.

The most accurate model, given a particular set of data, is the one that has the lowest value of <|Df|>, and there are strong indications that the value of <|Df|> is highly correlated (correlation coefficient close to +1) with that of the free R-factor, Rfree [1, 2]. Therefore, the most accurate model is the one with the lowest value of Rfree.

The level of detail at which one can describe a model is dictated by the information contents (determined by quantity and quality) of the data. For this reason, few people would be tempted to refine anisotropic temperature factors at 2.5Å. On the other hand, many people do refine individual isotropic temperature factors at 3Å or worse resolution, which in most cases amounts to over-fitting of the data. Here, Rfree can be used to decide if the increased level of detail (e.g., when going from one temperature factor per residue to individual isotropic Bs) is warranted by the data: if refinement of individual Bs leads to a reasonable drop in Rfree, the new model apparently gives a better description of the data; if it doesn't, refining individual Bs over-fits the data.

Degrees of "wrongness".

• completely wrong model or sub-unit: one in which essentially the entire chain has been traced incorrectly.
  
• partly wrong main-chain connectivity: usually due to incorrect connections between secondary-structure elements.
  
• out-of-register error: there is a frame shift for (usually) a small part of the sequence and density.
  
• locally poor model: either due to bad building or to insufficient data.
  
• incorrect side-chain conformations.
  
• incorrect peptide orientations.
  

• spacegroup error.
  
• sequence errors: either introduced at the sequencing stage or as a trivial typo.
  

• over-fitted models: by refining far more parameters than is warranted, the conventional R-factor can be reduced to almost arbitrarily low values, at the expense of a globally worse model. Favourite "tricks" to achieve this include: unrestrained refinement of NCS-related molecules, individual temperature factors at low resolution, fantasised waters, refined occupancies and alternative conformations at medium or low resolution.
  

How NOT to judge a model.

Judging from Table I, and using the conventional ideas as to what constitutes a "well-refined model", model "X" looks quite good, whereas model "Y" has a high R-factor and average temperature factor, indicating that there might be something wrong with it.

In fact, model "Y" is the structure of cellular retinoic-acid-binding protein (CRABP) type I [7]. The R-factor may seem high, but the structure was refined by minimising Rfree (to get the most accurate model), and by minimising the difference between R and Rfree (to minimise over-fitting). The temperature factors are high because the quality of the data was less than fantastic (the effective resolution is ~3.2Å), and partly because the structure was refined with strictly constrained two-fold NCS (see the discussion in [7]).

Model "X" is a related protein, CRABP type II [7], which was originally solved at 1.8Å. However, the correct structure was then intentionally traced backwards, and the resulting model was refined using data out to 3Å, to yield model "X" ... [8] Note that this means that the Brändén & Jones 25% R-factor threshold has been broken (this held that a refinement that stalled at an R-factor >0.25 should make "alarm bells ring").

In other words, the "conventional quality indicators" listed in Table I are not even capable of discriminating between a correct and a backward-traced protein structure ! In the following we shall encounter a number of quality indicators that do a much better job, in particular when they are used in combination (since a good model makes sense in all respects).

Making better models.

• there is only protein (no water, no ligand, etc.);
  
• the geometry is near-ideal;
  
• all NCS-related molecules are identical;
  
• there is only an overall temperature factor.
  

Every refinement cycle, except perhaps the few last ones, should involve high-temperature (4000 K) simulated annealing (SA) [10]. This removes (most of the) model bias and ensures that a large part of conformational space is sampled. It also indicates how "robust" the model is: well-defined parts of the structure do not suffer from high-temperature SA, except for the odd side chain.

As the model becomes better and more complete, it can be made more precise (i.e., detailed). For example, the ligand or co-factor can be included in the model, water molecules can be added, the NCS constraints can be replaced by restraints, and temperature factors can be refined for groups of atoms (e.g., two Bs per residue) or perhaps even individual atoms. However, one should realise that each of these steps increases the number of degrees of freedom, and thereby the potential for the refinement program to adjust these parameters in order to model noise and to mask errors in the structure. Undoubtedly atoms have individual (anisotropic) temperature factors, and NCS-related molecules display small differences, but the question one should ask is if the data is of sufficient quality and quantity to actually model these phenomena. At anything worse than atomic resolution, Rfree appears to be the only statistic that can actually tell if an increase in the precision of the model is warranted by the information contents of the data, i.e. if it constitutes an improved model for your particular dataset, or if the refinement program has merely used the additional freedom to reduce the conventional R-factor by making the model worse.

What if you don't ?

A case in point is the structure of chloromuconate cycloisomerase [12] (PDB code 1CHR). This structure was solved in spacegroup I4 with two-fold NCS. The model was refined against 3Å data without NCS-constraints, with individual temperature factors, with alternative conformations for some residues, and without Rfree. "Significant differences [...] at the active site" were found between the two NCS-related molecules, and their RMSD was 0.86Å on Ca atoms, and 1.5Å on all atoms. Closer inspection of the model and the data, however, revealed that the actual spacegroup is I422, without NCS [3]. In addition, it was found that a stretch of ~25 residues was out-of-register in the original model [3]. Both errors were masked by the refinement program: since there were ~1.5 times as many degrees of freedom as there were reflections, the wrong model in the wrong spacegroup still had a conventional R-factor of 0.195. Again this shows that the conventional R-factor is rather meaningless at worse than atomic resolution.

The major lessons to be learned from this are:

• (1) with conservative models (considerably more observations than degrees of freedom) it is difficult to go wrong, whereas with liberal models (more degrees of freedom than observations) one is almost bound to over-fit and thereby introduce and/or mask errors;
  
• (2) if crystallographically related (and therefore identical) molecules can be "refined" to an RMS positional difference of 1.5Å (provided the resolution is low enough), one has to wonder how different NCS-related molecules really are. We submit that most of the reported differences at medium and low resolution (>2Å) are over-estimates due to over-fitting. Our present "guesstimate" of realistic differences between NCS-related molecules is ~0.2-0.3Å [11], but more examples of structures with NCS solved at very high resolution are needed. One can only tell if NCS-related molecules are different if one refines them properly, i.e. starting with constraints, then refining with restraints (and checking if Rfree drops), and finally, perhaps, without restraints (and monitoring Rfree). If one starts off refining without restraints, "observed" differences are nothing but a self-fulfilled prophecy. In some cases, domain movements occur, making RMSD-type comparisons meaningless. But even in those cases one would still expect the "law of conservation of secondary structure" to apply, i.e. corresponding residues outside the hinge region(s) should have very similar main-chain dihedral angles.

Judging a paper.

• data quality and quantity: what was Rmerge, the completeness, the multiplicity and the strength of the data, both overall and in the highest resolution shell ? Have the authors collected a real 1.8Å dataset, or have they mostly collected indices between 2.2 and 1.8Å ?
  
• refinement protocol: have the authors carefully designed a refinement protocol suitable to their particular problem ? If Rfree has not been used throughout the refinement, there is no guarantee that the model is even remotely correct. If no mention is made of grouped temperature factors or NCS-constraints, one may assume that these have not been used and, in particular (but not exclusively) at low resolution, this will have introduced artificial differences between the NCS-related molecules and generally made the model worse than necessary.
  
• model quality: what have the authors done to make sure (and convince the reader) that the model is essentially correct ? Does the Ramachandran plot look normal ? Does the temperature-factor plot show regions with consistently high values ? Does the model look like a protein (i.e., in terms of rotamers, peptide orientation, etc.) ? If there is NCS, are the molecules similar (RMSD and RMS delta-B on all atoms and on core Ca atoms; delta-Phi and delta-Psi) ?
  

 a - calculated with LSQMAN (GJK & TAJ, unpublished program)
 b - calculated with ProCheck [16]
 c - calculated with O [5]
 d - calculated with What If [17]; this measures how (un)usual the
     ensemble of neighbouring protein atoms is for every group of
     atoms in the protein; this may discriminate against DNA-binding
     proteins (2GDA and 1GDC)
 e - many hydrogen bonds are flagged as bad contacts
 f - defined as residues having an RSC-fit value > 1.5 Å
 g - defined as residues having a pep-flip value > 2.5 Å
 h - this is a kringle domain (i.e., no a-helices or b-strands)

Judging coordinates.

• BACK: the backward-traced structure of CRABP type II (model "X" in Table I; [7]);
  
• ASGL: the largely incorrect structure of asparaginase/glutaminase [13];
  
• 1CHR: the original model of chloromuconate cycloisomerase [12], which was solved in the wrong spacegroup and had ~7% of its residues out-of-register [3];
  
• 1PMK: the structure of a plasminogen kringle domain [14], which is an example of a traditionally refined model at reasonably high resolution;
  
• two NMR structures (of the glucocorticoid receptor's DNA-binding domain [15]) have been included for comparison, the "final" energy-minimised model (1GDC) and model 24 of the ensemble of structures (2GDA);
  
• 1CBR: CRABP I (model "Y" in Table I; [7]) has been included as an example of a conservatively refined low-resolution model.
  

• (1) Temperature factors: the model, the average temperature factor and the RMS delta-Bs of bonded atoms.
  
• (2) NCS: RMS positional and temperature-factor differences between NCS-related atoms, differences in the main-chain dihedral angles.
  
• (3) Ramachandran plot: the percentage of residues in the four types of area as defined by ProCheck [16].
  
• (4) Secondary structure: the percentage of residues in helices and strands.
  
• (5) Geometry and stereo-chemistry: some of the properties calculated by ProCheck, the percentage of residues with non-rotamer side chains and unusual peptide orientations, and the overall G-factor.
  
• (6) Directional atomic-contact analysis score: this measures how (un)usual the ensemble of neighbouring protein atoms is for every group of atoms in the protein [17].
  

An important conclusion is that an essentially correct model scores well on basically all tests (i.e., makes sense in all respects), a partly incorrect model scores poor on a few tests, and a grossly wrong model scores poor on almost all tests (other than the conventional R-factor and RMSD values). The same is true, by the way, at the residue level: problematic regions tend to score poor on a number of different criteria (temperature factors, Ramachandran plot, peptide orientation, side-chain conformation, real-space R-factor, etc.).

Judging made easy.

 a - calculated with LSQMAN (GJK & TAJ, unpublished program)
 b - calculated with ProCheck [16]
 c - calculated with O [5]
 d - calculated with What If [17]
 e - many hydrogen bonds are flagged as bad contacts
 f - defined as residues having an RSC-fit value > 1.5 Å
 g - defined as residues having a pep-flip value > 2.5 Å

The ultimate test.

• 1GUH: 2.6Å structure of glutathione S-transferase (GST) A1-1 [19] refined with strict four-fold NCS;
  
• ALEX: a non-isomorphous 2Å structure of the same protein refined with restrained two-fold NCS [20];
  
• 9RUB: a 2.6Å structure of rubisco with two-fold NCS [21];
  
• 5RUB: the same structure, solved at 1.7Å [22].
  

In practice, one often encounters situations in which a structure is first solved at high resolution. This structure is then used to solve the structures of mutants or complexes for which only low-resolution datasets are available. A dangerous mistake is to use the same refinement protocol that was used for the high-resolution refinement for the low-resolution structures (e.g., no NCS restraints, individual temperature factors). The only way in which such structures can be refined properly is by (a) using a conservative refinement strategy, (b) using weak harmonic restraints to keep the atoms near their high-resolution positions unless there is a strong driving force in the data to change them, and (c) monitoring Rfree from the very start of the refinement. This approach has successfully been applied in the refinement of a complex of Candida antarctica lipase B at 2.5Å resolution, starting from a 1.5Å model [23, 24].

Acknowledgments.

References.

1. A.T. Brünger, Nature 355, 472 (1992).
   
2. A.T. Brünger, Acta Cryst. D49, 24 (1993).
   
3. G.J. Kleywegt & T.A. Jones, "A more correct crystal structure of chloromuconate cycloisomerase", to be published.
   
4. C.I. Brändén & T.A. Jones, Nature 343, 687 (1990).
   
5. T.A. Jones, J.Y. Zou, S.W. Cowan, & M. Kjeldgaard, Acta Cryst. A47, 110 (1991).
   
6. A.T. Brünger, "X-PLOR: a system for crystallography and NMR", Yale University, New Haven , CT, 1990.
   
7. G.J. Kleywegt, T. Bergfors, H. Senn, P. Le Motte, B. Gsell, K. Shudo, & T.A. Jones, Structure 2, 1241 (1994).
   
8. G.J. Kleywegt & T.A. Jones, "Refinement of low-resolution structures", to be published.
   
9. J.Y. Zou & S.L. Mowbray, Acta Cryst. D50, 237 (1994).
   
10. A.T. Brünger & A. Krukowski, Acta Cryst. A46, 585 (1990).
    
11. G.J. Kleywegt & T.A. Jones, "Maltreatment of non-crystallographic symmetry", to be published.
    
12. H. Hoier, M. Schlömann, A. Hammer, J.P. Glusker, H.L. Carrell, A. Goldman, J.J. Stezowski, & U. Heinemann, Acta Cryst. D50, 75 (1994).
    
13. H.L. Ammon, I.T. Weber, A. Wlodawer, R.W. Harrison, G.L. Gilliland, K.C. Murphy, L. Sjölin, & J. Roberts, Proc. Natl. Acad. Sci. USA 263, 150 (1988);
    J. Lubkowski, A. Wlodawer, D. Hosset, I.T. Weber, H.L. Ammon, K.C. Murphy, & A.L. Swain, Acta Cryst. D50, 826 (1994).
    
14. K. Padmanabhan, T.P. Wu, K.G. Ravichandran, & A. Tulinsky, Prot. Sci. 3, 898 (1994).
    
15. H. Baumann, K. Paulsen, H. Kovacs, H. Berglund, A.P.H. Wright, J.A. Gustafsson, & T. Härd, Biochemistry 32, 13463 (1993).
    
16. R.A. Laskowski, M.W. MacArthur, D.S. Moss, & J.M. Thornton, J. Appl. Cryst. 26, 283 (1993).
    
17. G. Vriend & C. Sander, J. Appl. Cryst. 26, 47 (1993).
    
18. R.A. Engh & R. Huber, Acta Cryst. A47, 392 (1991).
    
19. I. Sinning, G.J. Kleywegt, S.W. Cowan, P. Reinemer, H.W. Dirr, R. Huber, G.L. Gilliland, R.N. Armstrong, X. Ji, P.G. Board, B. Olin, B. Mannervik, & T.A. Jones, J. Mol. Biol. 232, 192 (1993).
    
20. A.D. Cameron, et al., & T.A. Jones, "Structure refinement and analysis of human alpha class glutathione S-transferase A1-1, in the apo form and in complexes with ethacrynic acid and its glutathione conjugate", to be published.
    
21. T. Lundqvist & G. Schneider, J. Biol. Chem. 266, 12604 (1991).
    
22. G. Schneider, Y. Lindqvist, & T. Lundqvist, J. Mol. Biol. 211, 989 (1990).
    
23. G.J. Kleywegt & T.A. Jones, "Good model-building and refinement practice", to be published.
    
24. J. Uppenberg, et al., & T.A. Jones, "Crystallographic and molecular dynamics studies of lipase B from Candida antarctica reveal a stereo-specificity pocket for secondary alcohols", to be published.