Comparison.exclude.unshared_tokens
Comparison.exclude.unshared_tokens ¶
Removes the tokens that do not appear in both of the corpora being compared.
Tokens are excluded from both corpora, before they are scored, so the frequencies the scoring measure works with are normalized over the remaining vocabulary.
Example
import wordlevel as wl
cl = wl.Catalog.from_dict_of_freqs(
{
"before": {"the": 30, "good": 12, "bad": 8, "happy": 6, "sad": 4, "calm": 5},
"after": {"the": 25, "good": 5, "bad": 7, "happy": 3, "sad": 9, "angry": 6},
}
)
cl = cl.with_comparisons(
wl.comp("before", "after").exclude.unshared_tokens().score.kullback_leibler_divergence()
)
Warning
Excluding tokens changes the normalized frequencies of every token that remains.
A token that only one corpus uses can be a stark difference between them. Prefer using a measure that can handle these absences, rather than excluding them entirely.
Returns:
| Type | Description |
|---|---|
Comparison
|
A new |