Academia.eduAcademia.edu

Figure from:Guided Generative Protein Design using Regularized Transformers

See full PDFdownloadDownload figure
We first examine representation smoothness with respect to fitness and sequence using amino acid sequence directly, labeled as Sequence’ in Table 2. As expected, this representation possesses the highest smoothness with respect to sequence however it’s smoothness with respect to fitness is less than that of other approaches. This likely due to the often tangled relationship between mutational distance and fitness desribed in Section 2.2. Furthermore, models which have trained to predict fitness produce a smoother representation with res pect to fitness than models trained solely on reconstruction. The smoothing effect of the fitness prediction task can be readily observed in Figure 4, where latent encodings from four models are visualized using principle component analysis (PCA) and PHATE Moon et al. [2017]. With the ablations performed in ReLSO, it is observed that removal of the interpolation sampling regularization (ReLSO (neg)) reduces the smoothness with respect to sequence, This effect is also observed in Figure 5 w variation in sequence change when the interpolation sam here walks in latent space possess greater pling regularization is removed. Lastly, we also compare to the protein representations learned by the pre-trained TAPE transformer model from Rao et al. [2019] which was trained on Pfam El-Gebali et al. [2019]. Table 2: Quantification of latent space ruggedness, described in Section 5.1 in Table 2. Ruggedness values with respect to fitness (Ay) and sequence (A;). The average of those two values \ = (A + A;)/2 is also reported.

Table 2 We first examine representation smoothness with respect to fitness and sequence using amino acid sequence directly, labeled as Sequence’ in Table 2. As expected, this representation possesses the highest smoothness with respect to sequence however it’s smoothness with respect to fitness is less than that of other approaches. This likely due to the often tangled relationship between mutational distance and fitness desribed in Section 2.2. Furthermore, models which have trained to predict fitness produce a smoother representation with res pect to fitness than models trained solely on reconstruction. The smoothing effect of the fitness prediction task can be readily observed in Figure 4, where latent encodings from four models are visualized using principle component analysis (PCA) and PHATE Moon et al. [2017]. With the ablations performed in ReLSO, it is observed that removal of the interpolation sampling regularization (ReLSO (neg)) reduces the smoothness with respect to sequence, This effect is also observed in Figure 5 w variation in sequence change when the interpolation sam here walks in latent space possess greater pling regularization is removed. Lastly, we also compare to the protein representations learned by the pre-trained TAPE transformer model from Rao et al. [2019] which was trained on Pfam El-Gebali et al. [2019]. Table 2: Quantification of latent space ruggedness, described in Section 5.1 in Table 2. Ruggedness values with respect to fitness (Ay) and sequence (A;). The average of those two values \ = (A + A;)/2 is also reported.

Related Figures (10)

Figure 1: Overall flow of information in ReLSO. (Top box) Foundational to ReLSO is the idea that protein sequences can be mapped to a latent fitness landscape, which is maintains better properties than the traditional sequence-level fitness landscape. In ReLSO, we use a transformer-based encoder to map sequences to point in well-organized latent space produced after training. (Bottom box) To encourage latent fitness landscapes amenable to latent space optimization, ReLSO incorporates several regularizations. Critical among these regularizations are negative sampling at the fitness prediction head hg and interpolative sampling at the sequence generation head gg
Figure 1: Overall flow of information in ReLSO. (Top box) Foundational to ReLSO is the idea that protein sequences can be mapped to a latent fitness landscape, which is maintains better properties than the traditional sequence-level fitness landscape. In ReLSO, we use a transformer-based encoder to map sequences to point in well-organized latent space produced after training. (Bottom box) To encourage latent fitness landscapes amenable to latent space optimization, ReLSO incorporates several regularizations. Critical among these regularizations are negative sampling at the fitness prediction head hg and interpolative sampling at the sequence generation head gg
Figure 2: An inherent weakness of a joint-training set-up is that a monotonic function is learned, which lacks any stopping criterion when used for latent space optimization. To address this, we reshape the fitness function such that the global maxima lies in/near the training data. Here we train the ReLSO architecture with a 2-dimensional latent encoding on the GIFFORD dataset and then visualize the learned fitness prediction function by overlaying predicted fitness values as well as model-derived fitness gradients. Shown are the effects of the latent encoding magnitude weighting 7 and the type of regressor network used on the overall shape of the fitness function An important consequence of the joint training set-up is that the latent code is updated during each train step with gradient signals from both sequence and fitness information. The resulting z encoding is thereby induced to resolve the two pieces of information. After model convergence, the latent space is endowed with a strong sequence-fitness association which we leverage here for latent space optimization.
Figure 2: An inherent weakness of a joint-training set-up is that a monotonic function is learned, which lacks any stopping criterion when used for latent space optimization. To address this, we reshape the fitness function such that the global maxima lies in/near the training data. Here we train the ReLSO architecture with a 2-dimensional latent encoding on the GIFFORD dataset and then visualize the learned fitness prediction function by overlaying predicted fitness values as well as model-derived fitness gradients. Shown are the effects of the latent encoding magnitude weighting 7 and the type of regressor network used on the overall shape of the fitness function An important consequence of the joint training set-up is that the latent code is updated during each train step with gradient signals from both sequence and fitness information. The resulting z encoding is thereby induced to resolve the two pieces of information. After model convergence, the latent space is endowed with a strong sequence-fitness association which we leverage here for latent space optimization.
Figure 3: Visualization of optimization paths in GIFFORD (A) Optimization trajectories of all 30 low-fitness seed sequences from the GIFFORD dataset from gradient-free methods (B) Optimization paths taken by gradient ascent converge to a high-fitness region of latent space. This is a result of the underlying regularized fitness function learned by ReLSO (C) For the optimization of a single seed, paths taken by gradient-free methods show a less-efficient progression to the data-driven global fitness maxima.
Figure 3: Visualization of optimization paths in GIFFORD (A) Optimization trajectories of all 30 low-fitness seed sequences from the GIFFORD dataset from gradient-free methods (B) Optimization paths taken by gradient ascent converge to a high-fitness region of latent space. This is a result of the underlying regularized fitness function learned by ReLSO (C) For the optimization of a single seed, paths taken by gradient-free methods show a less-efficient progression to the data-driven global fitness maxima.
Figure 4: A. Here we compare the three major representation methods for proteins sequences in this work, namely the amino acid sequence (green edge), latent encoding from an unsupervised autoen.- coder (blue edge), and a jointly-trained autoencoder (orange edge). B. Grids of the representations of sequences from four datasets (left to right: GIFFORD, GB1, GFP, TAPE) visualized by PCA and colored by their respective fitness values. C. Here we visualize the same represenations now using PHATE Moon et al. [2017], which visualizes multi-scale organization of the data. D. We use a smoothness metric which measure neighborhood-level variation of fitness values in latent space and compare the smoothness acroess the various approaches to protein representations in Table 2
Figure 4: A. Here we compare the three major representation methods for proteins sequences in this work, namely the amino acid sequence (green edge), latent encoding from an unsupervised autoen.- coder (blue edge), and a jointly-trained autoencoder (orange edge). B. Grids of the representations of sequences from four datasets (left to right: GIFFORD, GB1, GFP, TAPE) visualized by PCA and colored by their respective fitness values. C. Here we visualize the same represenations now using PHATE Moon et al. [2017], which visualizes multi-scale organization of the data. D. We use a smoothness metric which measure neighborhood-level variation of fitness values in latent space and compare the smoothness acroess the various approaches to protein representations in Table 2
Figure 5: Smoothness along 100 sampled walks between pairs of distant points in latent space. Each step x; is along a nearest neighbor search to a final latent point x. AFit denotes L1 distance between each step’s fitness and the final point’s fitness. Similarly, ASeq denotes Hamming distance of each step’s sequence to the final step’s sequence. Ablation of negative sampling or interpolative sampling effects the smoothness of latent space traversal 5.1 Assessing Representation Smoothness
Figure 5: Smoothness along 100 sampled walks between pairs of distant points in latent space. Each step x; is along a nearest neighbor search to a final latent point x. AFit denotes L1 distance between each step’s fitness and the final point’s fitness. Similarly, ASeq denotes Hamming distance of each step’s sequence to the final step’s sequence. Ablation of negative sampling or interpolative sampling effects the smoothness of latent space traversal 5.1 Assessing Representation Smoothness
Table 3: Optimization of protein fitness. Here we evaluate the performance of three categories of optimization methods. The first three entries can be understood as optimization within latent space using local search approach. Latent space Metropolis Hastings MCMC searches pockets of latent space whereas hill climbing (HC) and stochastic hill climbing (SHC) use the locations of nearby points for directions to move in. The second category made up of Metropolis Hastings MCMC (MCMC Seq) and directed evolution (DE) are optimization approaches which operate on the sequence directly. The final category is a latent space optimization approach which uses a gradient-guided search process, gradient ascent (GA). We observe generally greater performance with this third class of method in the proposed evaluation critera.
Table 3: Optimization of protein fitness. Here we evaluate the performance of three categories of optimization methods. The first three entries can be understood as optimization within latent space using local search approach. Latent space Metropolis Hastings MCMC searches pockets of latent space whereas hill climbing (HC) and stochastic hill climbing (SHC) use the locations of nearby points for directions to move in. The second category made up of Metropolis Hastings MCMC (MCMC Seq) and directed evolution (DE) are optimization approaches which operate on the sequence directly. The final category is a latent space optimization approach which uses a gradient-guided search process, gradient ascent (GA). We observe generally greater performance with this third class of method in the proposed evaluation critera.
Figure 6: Optimization efficiency. (A) Here the final step fitness values of all 30 seeds are plotted, separated by dataset. We observe that gradient ascent is able to generate a greater number of high-fitness protein sequences compared to other methods. In the case of GFP, we observe that all optimized candidates of gradient ascent reside within the high-flourescence mode of the dataset. (B) The evolution of fitness values across GIFFORD seed sequences show a wide spread of efficiency. As some methods label multiple sequences during a single optimization step (e.g. HC, directed evolution) some lines do not each across the entire plot. Gradient ascent’s quick convergence to high-fitness is highlighted by the inclusion of a dotted vertical line at optimization step 15. To visualize the path taken by each optimization method, we co-embed the sequences encountered by each method into the latent space of the ReLSO model. We observe that gradient ascent is able to efficiently move through the latent space to a high-fitness region and as a result of the reshaped model fitness landscape, optimization trajectories converge near or at peak fitness, as shown in Figure 3B. This behavior keeps optimization trajectories from drifting into regions of latent space that the model has little information about and thus would likely make false predictions about. This is in contrast
Figure 6: Optimization efficiency. (A) Here the final step fitness values of all 30 seeds are plotted, separated by dataset. We observe that gradient ascent is able to generate a greater number of high-fitness protein sequences compared to other methods. In the case of GFP, we observe that all optimized candidates of gradient ascent reside within the high-flourescence mode of the dataset. (B) The evolution of fitness values across GIFFORD seed sequences show a wide spread of efficiency. As some methods label multiple sequences during a single optimization step (e.g. HC, directed evolution) some lines do not each across the entire plot. Gradient ascent’s quick convergence to high-fitness is highlighted by the inclusion of a dotted vertical line at optimization step 15. To visualize the path taken by each optimization method, we co-embed the sequences encountered by each method into the latent space of the ReLSO model. We observe that gradient ascent is able to efficiently move through the latent space to a high-fitness region and as a result of the reshaped model fitness landscape, optimization trajectories converge near or at peak fitness, as shown in Figure 3B. This behavior keeps optimization trajectories from drifting into regions of latent space that the model has little information about and thus would likely make false predictions about. This is in contrast
Figure 7: Latent space visualization of high-fitness generated variants from the Gifford dataset. (A) Principal component analysis of the original training dataset model predicted embedding for each sequence. (B) Principal component addition of the 11,061 generated sequences onto the original latent space (yellow = high binding affinity/fitness, blue = low binding affinity/fitness). (C) Histogram distribution of fitness values for the original training dataset. (D) Histogram distribution of fitness values of the predicted fitness values from the generated sequences overlapped onto the original dataset distribution for visualization (black = original, red = generated) Upon interpolation of the high fitness sequences, each of 11,061 sequences was independently run through the trained model for both binding affinity and embedding predictions. The embedding of each sequence was obtained and plotted onto the latent space in the context of the entire training dataset. Interestingly, the generated sequences localized to a single high fitness region of the latent space. This indicated a denser sampling and fitness prediction of the local space using generated sequences that were not part of the initial dataset. Additionally, the generated sequence abundance appeared to be associated with increasing fitness as shown in Figure 7, which was in line with the learned latent space embedding from the initial training data. These generative results suggest
Figure 7: Latent space visualization of high-fitness generated variants from the Gifford dataset. (A) Principal component analysis of the original training dataset model predicted embedding for each sequence. (B) Principal component addition of the 11,061 generated sequences onto the original latent space (yellow = high binding affinity/fitness, blue = low binding affinity/fitness). (C) Histogram distribution of fitness values for the original training dataset. (D) Histogram distribution of fitness values of the predicted fitness values from the generated sequences overlapped onto the original dataset distribution for visualization (black = original, red = generated) Upon interpolation of the high fitness sequences, each of 11,061 sequences was independently run through the trained model for both binding affinity and embedding predictions. The embedding of each sequence was obtained and plotted onto the latent space in the context of the entire training dataset. Interestingly, the generated sequences localized to a single high fitness region of the latent space. This indicated a denser sampling and fitness prediction of the local space using generated sequences that were not part of the initial dataset. Additionally, the generated sequence abundance appeared to be associated with increasing fitness as shown in Figure 7, which was in line with the learned latent space embedding from the initial training data. These generative results suggest
Table 4: Task performance across the two tasks. Task | has the model predict sequence from laten space whereas Task 2 requires the model to predict fitness In our model set-up, we train on two tasks - a reconstruction task and a fitness prediction task. The first of requires that the model be able to generate sequences from the low-dimensional latent encoding. This allows the model to be generative, which is crucial in the domain of biomolecules as complete enumeration of sequence space is infeasible. We train the model using a Cross-Entropy loss across the possible amino acid identities of each position in the sequence. Here we report accuracy as well as perplexity to assess model performance in this task in Table 4.
Table 4: Task performance across the two tasks. Task | has the model predict sequence from laten space whereas Task 2 requires the model to predict fitness In our model set-up, we train on two tasks - a reconstruction task and a fitness prediction task. The first of requires that the model be able to generate sequences from the low-dimensional latent encoding. This allows the model to be generative, which is crucial in the domain of biomolecules as complete enumeration of sequence space is infeasible. We train the model using a Cross-Entropy loss across the possible amino acid identities of each position in the sequence. Here we report accuracy as well as perplexity to assess model performance in this task in Table 4.
Figure 8: (Top) GB1 protein structure with overalayed attention. Here we visualize the pairs of residues with the highest interactions as determined by the attention maps in our model. Here we see a preference for residue 43. (Bottom) GFP protein structure with overalayed attention. Differential attention between ReLSO and the purely unsupervised AE model may indicates relations important for function
Figure 8: (Top) GB1 protein structure with overalayed attention. Here we visualize the pairs of residues with the highest interactions as determined by the attention maps in our model. Here we see a preference for residue 43. (Bottom) GFP protein structure with overalayed attention. Differential attention between ReLSO and the purely unsupervised AE model may indicates relations important for function