There are quite a few tools for similarity threshold-based filtering of DNA or protein sequences. The best choice depends on whether you need exact identity clustering, fast deduplication, homology reduction, or large-scale metagenomic processing.
| Tool | DNA | Protein | Typical similarity measure | Best use case |
|---|
| CD-HIT | ✓ | ✓ | Global sequence identity | Fast redundancy reduction |
| MMseqs2 | ✓ | ✓ | Sequence identity/alignment | Large datasets, clustering, filtering |
| VSEARCH | ✓ | Limited | Identity-based | Amplicon and nucleotide dereplication |
| USEARCH | ✓ | ✓ | Identity | OTU clustering, dereplication |
| Linclust (MMseqs2) | ✓ | ✓ | Approximate identity | Very large protein databases |
| BLAST + custom filtering | ✓ | ✓ | Identity, coverage, E-value | Flexible pairwise filtering |
| DIAMOND | Limited | ✓ | Protein homology | Fast protein similarity searches |
| Mash | ✓ | — | MinHash distance | Genome pre-filtering |
| Sourmash | ✓ | ✓ (translated) | Sketch similarity | Massive genome comparisons |
1. CD-HIT
One of the most widely used redundancy-reduction tools.
Pros
- Extremely fast
- Simple identity threshold (
-c)
- Supports proteins and nucleotides
- Standard for creating nonredundant datasets
Example:
cd-hit -i proteins.fasta -o proteins90 -c 0.90
Keeps representatives with ≥90% identity collapsed together.
Typical thresholds:
- 0.90
- 0.95
- 0.98
- 1.00 (exact duplicates)
2. MMseqs2
Arguably the current state-of-the-art for large sequence collections.
Advantages:
- Much faster than BLAST
- More sensitive than CD-HIT
- Handles millions to billions of sequences
- Rich filtering options
Example clustering:
mmseqs easy-cluster input.fasta output tmp --min-seq-id 0.9
Supports additional filters such as
- minimum identity
- alignment coverage
- E-value
- alignment length
Very popular in modern protein database construction.
3. Linclust
Part of MMseqs2.
Designed specifically for huge datasets.
Can cluster:
- UniProt-scale databases
- Metagenomes
- Billions of proteins
Complexity is close to linear rather than quadratic.
4. VSEARCH
Open-source replacement for USEARCH.
Excellent for nucleotide sequences.
Example:
vsearch --cluster_fast reads.fasta \
--id 0.97 \
--centroids clustered.fasta
Widely used for:
- 97% OTU clustering
- dereplication
- amplicon analysis
5. USEARCH
Still widely used despite licensing restrictions.
Offers:
- clustering
- dereplication
- chimera removal
- abundance filtering
Historically important in microbial ecology.
6. BLAST
BLAST itself does not cluster sequences, but you can filter hits using criteria such as:
- percent identity
- query coverage
- subject coverage
- E-value
- bit score
Typical workflow:
blastp
↓
identity >90%
coverage >80%
E-value <1e-20
↓
remove redundant hits
Useful when precise pairwise relationships matter.
7. DIAMOND
For protein datasets.
Advantages:
- Thousands of times faster than BLASTP
- Similar filtering options
- Excellent for metagenomic annotation
Typical filters:
- identity
- coverage
- E-value
- bit score
8. Mash
Rather than aligning sequences, Mash compares MinHash sketches.
Good for:
- bacterial genomes
- rapid genome screening
- removing near-identical genomes before alignment
Much faster than alignment-based methods.
9. Sourmash
Uses MinHash sketches similar to Mash.
Useful for:
- genome collections
- metagenomes
- protein signatures
- scalable similarity searches
Which tool should you choose?
| Goal | Recommended tool |
|---|
| Remove duplicate protein sequences | CD-HIT or MMseqs2 |
| Cluster millions of proteins | MMseqs2 Linclust |
| Remove highly similar DNA sequences | VSEARCH or CD-HIT-EST |
| Create nonredundant training datasets | CD-HIT or MMseqs2 |
| Metagenomic protein filtering | MMseqs2 or DIAMOND |
| Genome-level similarity | Mash or Sourmash |
| Flexible identity + coverage filtering | BLAST or DIAMOND |
Common similarity thresholds
| Threshold | Typical application |
|---|
| 100% | Exact duplicate removal |
| 99% | Nearly identical sequences |
| 95% | Protein family reduction; bacterial ANI-related workflows (not directly equivalent to ANI) |
| 90% | General redundancy reduction |
| 70–80% | Broader protein family clustering |
| 30–40% | Remote homolog detection (usually with alignment/search tools rather than simple clustering) |
For most modern bioinformatics pipelines, MMseqs2 is often the preferred choice because it combines high speed, scalability, and flexible filtering. CD-HIT remains a strong option for straightforward redundancy reduction, especially for small to medium-sized datasets, while VSEARCH is a common choice for nucleotide amplicon workflows.
