Skip to content

Comparison.score.kullback_leibler_divergence

Comparison.score.kullback_leibler_divergence 

kullback_leibler_divergence(base: float = 2) -> Comparison

Calculates the Kullback-Leibler divergence (KLD) between the reference and comparison corpora.

The KLD is an asymmetric measure of how two corpora differ. From an information-theoretic perspective, it measures how much more information is needed to encode the comparison corpus when using the reference as the "codebook." It is only well-defined when all of the words in the comparison also appear in the reference. Otherwise, the KLD diverges to infinity.

Example 
import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").score.kullback_leibler_divergence())
Warning

Using this measure forces the comparison to be evaluated eagerly rather than lazily, because validating it requires computing over the full vocabulary.

The contribution \(\delta_\tau\) of a token \(\tau\) is:

\[ \delta_\tau = p_\tau^{(C)} \log \frac{1}{p_\tau^{(R)}} - p_\tau^{(C)} \log \frac{1}{p_\tau^{(C)}} \]

where \(p_\tau^{(R)}\) and \(p_\tau^{(C)}\) are the normalized frequencies of token \(\tau\) in \(R\) and \(C\) respectively, the reference and comparison corpora. The total divergence is always non-negative, but individual per-token contributions can be either sign. A positive contribution identifies a token that appears relatively more in the comparison corpus; a negative contribution, more in the reference. Because the total is non-negative, positive contributions reinforce the divergence and negative contributions offset it.

Reference

For more details, see the following paper:

Gallagher, R. J., Frank, M. R., Mitchell, L., Schwartz, A. J., Reagan, A. J., Danforth, C. M., & Dodds, P. S. (2021). Generalized word shift graphs: a method for visualizing and explaining pairwise comparisons between texts. EPJ Data Science, 10(1), 4.

See Also
Score Description
Comparison.score.jensen_shannon_divergence Scores corpora using a bounded, symmetric, entropy-based divergence

Parameters:

Name Type Description Default

base

float

Base of the logarithms used to calculate the KLD

 2

Returns:

Type Description
Comparison

A new Comparison scored by the Kullback-Leibler divergence

Raises:

Type Description
ValueError

If the comparison corpus contains any token absent from the reference corpus, which causes the KLD to be infinite. Filter to the intersection of tokens or use a symmetric measure like the Jensen-Shannon divergence

Source