Here's one way to think about the problem if we completely ignore existing architectures (EVOformer, MSA Transformer, AlphaFold, etc.) and instead ask:
What is the most information-preserving representation of an MSA as an evolutionary object?
A DNA MSA is not a sequence.
It is also not an image.
It is closer to a stochastic evolutionary process observed from many correlated samples.
Instead of tokenizing nucleotides, tokenize local evolutionary events.
Idea 1: Evolutionary Event Tokens (EET)
Suppose an MSA
Human ACTGTCA
Chimp ACTGTCA
Mouse ACTATTA
Dog ACTGTTA
Chicken GCTATTA
Normally we'd tokenize
A C T G T C A
This ignores the information contained in the column.
Instead each nucleotide is represented by
(current nucleotide,
column frequency,
ancestral probability,
mutation entropy,
gap history,
phylogenetic depth)
Example
Human position 5
A
↓
Token
{
base=A,
freq(A)=0.82,
entropy=0.22,
ancestral=A (0.94),
substitution_rate=0.01,
gap_prob=0.00
}
So every nucleotide becomes
E_i = embedding(
base,
freq,
entropy,
ancestral_prob,
branch_length,
gap,
codon_phase,
GC_context,
CpG_state
)
instead of
Embedding(A)
But we can go much further.
Idea 2: Context is a Distribution, not a Column
Current models usually compress one MSA column.
Instead build
Context(position i)
=
Joint distribution
P(column i-k ... i+k)
For every nucleotide
construct
Evolutionary neighborhood
radius = 8 columns
Compute
co-substitution matrix
16x16
Example
Position i
A
Neighbor j
G
Across species
A↔G changes together
Score = 0.91
Essentially
local evolutionary covariance tensor
becomes part of the token.
Idea 3: Encode Columns as Evolutionary Signatures
Each alignment column becomes
S_i
=
[
freq(A,C,G,T,-),
entropy,
transition/transversion ratio,
phylogenetic consistency,
Shannon diversity,
Gini impurity,
mutual information to nearby columns,
indel frequency,
estimated age,
branch-specific conservation
]
Now instead of nucleotide tokens
A
we have
Token
(A,
S_i)
Idea 4: Hypergraph Tokenization
This is probably the most interesting.
Treat the MSA as a hypergraph.
Vertices
(sequence,
position)
Edges
same column
Hyperedges
co-evolving positions
species lineage
shared indel
shared substitution history
Each nucleotide receives messages through
Column
↓
Phylogeny
↓
Covariation graph
↓
Motif graph
↓
Indel graph
before entering the Transformer.
So tokenizer outputs
latent graph tokens
instead of characters.
Idea 5: Evolutionary Compression Tokens
Imagine compressing an MSA.
Instead of storing every nucleotide
store
consensus
+
exceptions
Example
Consensus
ACTGCTA
Species deviations
Mouse
....A..
Dog
.....T.
Chicken
G..A...
Now tokenizer becomes
Consensus token
Deviation token
Gap token
Rare mutation token
This is similar to residual coding.
The encoder learns
consensus representation
Decoder reconstructs every species.
Huge reduction in redundancy.
Idea 6: Evolutionary Wavelets
Think of conservation like frequency.
Across neighboring positions compute
wavelet decomposition
Highly conserved
↓
low-frequency signal
Rapid mutation
↓
high-frequency signal
Token becomes
base
+
wavelet coefficients
This lets the network distinguish
AAAAAA
from
ACGTAC
without long attention.
Idea 7: Phylogenetic Attention Coordinates
Instead of positional encoding
use
(sequence_position,
branch_length,
LCA depth,
estimated ancestral state,
clade ID)
Embedding
Embedding
=
DNA embedding
+
Evolution embedding
For example
A
+
branch length =0.01
+
ancestor confidence=0.93
+
placental mammal
+
column entropy=0.12
Two identical nucleotides become different tokens depending on evolutionary history.
Idea 8: Learn a Latent Evolutionary Alphabet
This is probably the most radical.
Instead of
A,C,G,T
learn
4096 evolutionary symbols
using vector quantization.
Input
32-column MSA window
Encoder
↓
continuous latent
↓
VQ layer
↓
codebook index
Example
Window
↓
Code 812
Code 812 might correspond to
highly conserved exon
Another code
1043
might mean
rapidly evolving enhancer
Decoder reconstructs
entire MSA window.
Now the Transformer sees
812 1043 721 ...
instead of DNA.
This is analogous to language models learning words instead of letters.
Idea 9: Evolutionary Patch Tokens
Borrow from Vision Transformers.
Instead of
1 nucleotide = 1 token
tokenize
16 columns
×
32 species
Each patch
32 ×16 matrix
passes through a small encoder
↓
latent vector
↓
Transformer.
The local encoder learns
- substitution patterns
- lineage structure
- indels
- conservation
before global attention.
Idea 10: Factorized Encoder–Decoder Architecture (Evolutionary Latent Autoencoder)
Instead of reconstructing nucleotides directly, separate the latent representation into distinct evolutionary factors. This encourages disentanglement and gives the decoder richer, structured information.
Input MSA
│
▼
Patch Encoder
│
▼
Evolution Tokenizer
│
├──────────────┬───────────────┬──────────────┐
▼ ▼ ▼ ▼
Conservation Mutation Phylogeny Structure
Latent Latent Latent Latent
│ │ │ │
└──────────────┴───────────────┴──────────────┘
│
▼
Cross-scale Transformer
│
▼
Autoregressive / Masked Decoder
│
▼
Reconstruct complete MSA
Rather than one monolithic latent vector, each nucleotide (or patch) is represented by a tuple:
[
z_i = \left(z_i^{\text{cons}},;
z_i^{\text{mut}},;
z_i^{\text{phylo}},;
z_i^{\text{context}}\right)
]
where:
- (z^{\text{cons}}) captures conservation and selective constraint.
- (z^{\text{mut}}) models local mutation processes.
- (z^{\text{phylo}}) encodes lineage-specific evolutionary history.
- (z^{\text{context}}) summarizes neighboring co-evolution and sequence context.
The decoder learns to fuse these factors when reconstructing each aligned nucleotide, making the latent space more interpretable and potentially transferable to downstream tasks.
A More Radical Proposal: A Generative Evolutionary Program
Instead of asking the model to encode what nucleotides are present, ask it to encode the evolutionary program that generated them.
For each genomic window, infer a compact latent program:
Root (ancestral state)
│
▼
Substitution process
│
▼
Selection pressure field
│
▼
Insertion/deletion events
│
▼
Observed extant sequences
The encoder learns a latent representation of this generative process, and the decoder reconstructs the observed MSA by "executing" the inferred evolutionary program. This shifts the objective from memorizing aligned characters to modeling the stochastic dynamics of sequence evolution. Such a latent program could naturally support ancestral reconstruction, conservation prediction, simulation of future mutations, and transfer learning across species because it represents the underlying evolutionary mechanism rather than just its outcomes.
In essence, this reframes the MSA from a matrix of symbols into a compressed, executable description of evolutionary history. If successful, the latent variables become approximations of evolutionary forces (mutation, selection, drift, and lineage-specific events), providing both predictive power and biological interpretability.
