Information-theoretic metrics are increasingly used to analyze, compare, and curate datasets for training LLMs. They measure not just dataset size, but the amount, diversity, predictability, and uniqueness of the information contained in the data. The most commonly used metrics include:
| Metric | Measures | Typical Use |
|---|
| Shannon entropy | Average information content | Dataset diversity |
| Cross-entropy | Predictability under a model | Dataset quality, model fit |
| Perplexity | Exponential of cross-entropy | Dataset difficulty |
| KL divergence | Difference between distributions | Compare corpora |
| Jensen-Shannon divergence | Symmetric distribution difference | Domain similarity |
| Mutual information | Shared information between variables | Redundancy, feature analysis |
| Conditional entropy | Remaining uncertainty | Context dependence |
| Compression ratio | Effective entropy estimate | Deduplication, complexity |
| Effective vocabulary entropy | Lexical diversity | Corpus richness |
| n-gram entropy | Local sequence diversity | Style and repetitiveness |
Here's what each metric captures.
1. Shannon Entropy
The foundational measure:
[
H(X) = -\sum_x p(x)\log_2 p(x)
]
It quantifies the average information per symbol.
For LLM datasets it can be computed over:
- characters
- bytes
- words
- tokens
- n-grams
- documents
Higher entropy generally indicates:
- richer vocabulary
- less repetition
- greater linguistic diversity
Example:
Dataset A:
the cat sat
the cat sat
the cat sat
Dataset B:
quantum fields exhibit spontaneous symmetry breaking
Dataset B has much higher entropy.
2. Cross Entropy
Suppose one corpus defines distribution (P), another defines (Q).
Cross entropy is
[
H(P,Q)
]
It answers:
How surprising is corpus P when encoded using corpus Q?
This is widely used when comparing datasets.
Applications:
- Wikipedia vs Common Crawl
- Books vs Reddit
- Legal vs Medical
- English vs multilingual corpora
Lower cross entropy means the corpora are statistically similar.
3. Perplexity
Perplexity is
[
2^{H}
]
or
[
e^H
]
depending on the logarithm base.
It represents the effective number of equally likely next tokens.
Lower perplexity indicates:
- easier to predict
- more repetitive
- less information per token
Higher perplexity suggests:
- more varied language
- greater complexity
- broader vocabulary
Training datasets are often compared by measuring a reference model's perplexity on each.
4. KL Divergence
[
D_{KL}(P||Q)
]
Measures how much one distribution differs from another.
Applications include:
- comparing token frequencies
- comparing topic distributions
- comparing domain distributions
- detecting dataset shift
Example:
Wikipedia:
science
history
biology
Reddit:
lol
bro
meme
KL divergence would be large.
5. Jensen-Shannon Divergence
KL divergence has drawbacks:
- asymmetric
- can become infinite
Jensen-Shannon divergence addresses these issues:
[
JS(P,Q)
]
Properties:
- symmetric
- bounded
- easier to interpret
It is widely used for:
- corpus similarity
- language identification
- domain adaptation
- measuring overlap between datasets
6. Mutual Information
[
I(X;Y)
]
Measures shared information.
Applications in LLM datasets:
- token ↔ label
- token ↔ document
- word ↔ topic
- sentence ↔ metadata
High mutual information can reveal:
- duplicated documents
- boilerplate
- repeated templates
- highly correlated features
7. Conditional Entropy
[
H(Y|X)
]
Measures uncertainty remaining after observing context.
In language:
The capital of France is _____
Very low conditional entropy.
Versus
The next word could be ...
Higher conditional entropy.
Dataset comparison often examines average conditional entropy across domains.
8. Compression Ratio
Compression algorithms estimate entropy indirectly.
For example:
Highly compressible datasets tend to contain:
- duplicates
- repeated phrases
- templates
- boilerplate
Low compressibility often indicates:
- higher information density
- greater diversity
Modern deduplication pipelines often rely on compression-based similarity measures.
9. Vocabulary Entropy
Rather than just counting unique words, vocabulary entropy considers their frequency distribution.
A corpus with:
- 1 million unique words where one dominates
has lower entropy than one where usage is more evenly distributed.
This metric better reflects lexical richness than vocabulary size alone.
10. n-gram Entropy
Entropy can be computed over sequences:
This captures:
- syntactic diversity
- phrase repetition
- formulaic language
For example:
Legal documents often exhibit lower n-gram entropy due to standardized phrasing.
More advanced information-theoretic measures
Researchers also use:
- Normalized Compression Distance (NCD): Estimates similarity between datasets based on how well they compress together.
- Entropy rate: The average information produced by a stochastic process, accounting for long-range dependencies.
- Rényi entropy: A family of entropy measures that emphasize common or rare events depending on a parameter.
- Min-entropy: Focuses on the probability of the most likely event, useful for assessing worst-case unpredictability.
- Variation of Information (VI): An information-theoretic distance between clusterings or partitions, sometimes used in document organization analyses.
Metrics used in modern LLM data curation
Large-scale LLM training pipelines typically combine multiple information-theoretic signals with heuristic and model-based filters. Common components include:
- Token entropy
- Cross-entropy under a reference language model
- Perplexity
- KL or Jensen-Shannon divergence to monitor domain balance
- Compression ratio or locality-sensitive hashing for deduplication
- n-gram entropy to detect repetitive or templated content
- Mutual information or related measures to identify redundancy between documents
These metrics are usually supplemented with quality filters (e.g., language identification, toxicity detection, formatting checks) and semantic embedding-based methods.
Current research direction
Rather than evaluating datasets solely by their number of tokens, current work increasingly aims to quantify their effective information content. Active research areas include estimating:
- Information density: How much new information each token contributes.
- Novelty: How much a document differs from the rest of the corpus.
- Redundancy: The extent of repeated or near-duplicate content.
- Coverage: How well the dataset spans linguistic, topical, and stylistic distributions.
- Effective dataset size: The amount of unique learning signal after accounting for duplication and statistical dependence, often estimated using scaling laws and information-theoretic analyses.
The trend is toward viewing a training corpus as an information source whose utility depends on both its volume and the diversity and novelty of the information it provides.
