Skip to content

Comparison.score.shannon_entropy

Comparison.score.shannon_entropy 

shannon_entropy(
    base: float = 2,
    reference_score: float | Literal["mean"] | Expr = 0,
) -> Comparison

Calculates the difference in Shannon entropy between the reference and comparison corpora.

Example 
import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").score.shannon_entropy())

Shannon entropy is a measure of textual diversity and "surprise." This method takes the difference between the entropies of the reference corpus \(R\) and the comparison corpus \(C\):

\[ \Phi^{(C)} - \Phi^{(R)} = \sum_\tau p_\tau^{(C)} \log_b \frac{1}{p_\tau^{(C)}} - \sum_\tau p_\tau^{(R)} \log_b \frac{1}{p_\tau^{(R)}} \]

where \(p_\tau^{(R)}\) and \(p_\tau^{(C)}\) are the normalized frequencies of a token \(\tau\) in \(R\) and \(C\) respectively, and \(b\) is the logarithm base. The factor \(\log_b \frac{1}{p_\tau}\) is the "surprisal" of a token: the less frequent a token, the higher its surprisal.

Reference

For more details, see the following paper:

Gallagher, R. J., Frank, M. R., Mitchell, L., Schwartz, A. J., Reagan, A. J., Danforth, C. M., & Dodds, P. S. (2021). Generalized word shift graphs: a method for visualizing and explaining pairwise comparisons between texts. EPJ Data Science, 10(1), 4.

See Also
Score Description
Comparison.score.tsallis_entropy Scores corpora using generalized entropy

Parameters:

Name Type Description Default

base

float

Base of the logarithms used to calculate the entropy

 2

reference_score

float | Literal['mean'] | Expr

Baseline against which per-token contributions are interpreted. Defaults to 0. If "mean", uses the Shannon entropy of the reference corpus as the reference score

 0

Returns:

Type Description
Comparison

A new Comparison scored by the difference in Shannon entropies

Source