Comparison.score.shannon_entropy

Comparison.score.shannon_entropy ¶

shannon_entropy(
    base: float = 2,
    reference_score: float | Literal["mean"] | Expr = 0,
) -> Comparison

Calculates the difference in Shannon entropy between the reference and comparison corpora.

Shannon entropy is a measure of textual diversity and "surprise." If \(p_\tau\) is the normalized frequency of a token \(\tau\) in a corpus, then its Shannon entropy is \(\sum_\tau p_\tau \log \frac{1}{p_\tau}\). The factor \(\log \frac{1}{p_\tau}\) is known as the "surprisal" of a token: the less frequent a token, the higher its surprisal.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").score.shannon_entropy())

When the reference_score is 0, the contribution \(\delta_\tau\) of a token \(\tau\) is:

\[ \delta_\tau = p_\tau^{(C)} \log \frac{1}{p_\tau^{(C)}} - p_\tau^{(R)} \log \frac{1}{p_\tau^{(R)}} \]

where \(p_\tau^{(R)}\) and \(p_\tau^{(C)}\) are the normalized frequencies of token \(\tau\) in \(R\) and \(C\) respectively, the reference and comparison corpora.

Each token is interpreted by whether it reinforces or offsets the corpus-level difference in entropies \(\Phi^{(C)} - \Phi^{(R)}\). A contribution with the same sign as the overall difference reinforces it, making its magnitude larger; a contribution with the opposite sign offsets it, pulling the difference back toward zero. For example, if \(\Phi^{(C)} - \Phi^{(R)} > 0\) (the comparison's entropy is higher), positive contributions reinforce the difference and negative contributions offset it; the interpretation flips when \(\Phi^{(C)} - \Phi^{(R)} < 0\).

When the reference_score is non-zero, the contribution of the token is:

\[ \delta_\tau = \left( p_\tau^{(C)} - p_\tau^{(R)} \right) \left[ \frac{1}{2} \left( \phi_\tau^{(C)} + \phi_\tau^{(R)} \right) - \Phi^{(\text{ref})} \right] + \frac{1}{2} \left( p_\tau^{(C)} + p_\tau^{(R)} \right) \left( \phi_\tau^{(C)} - \phi_\tau^{(R)} \right) \]

where \(\Phi^{(\text{ref})}\) is the user-provided reference score, and

\[ \phi_\tau^{(C)} = \log \frac{1}{p_\tau^{(C)}} \, , \, \phi_\tau^{(R)} = \log \frac{1}{p_\tau^{(R)}} \]

Reference

For more details, see the following paper:

Gallagher, R. J., Frank, M. R., Mitchell, L., Schwartz, A. J., Reagan, A. J., Danforth, C. M., & Dodds, P. S. (2021). Generalized word shift graphs: a method for visualizing and explaining pairwise comparisons between texts. EPJ Data Science, 10(1), 4.

Name	Type	Description	Default
`base` ¶	`float`	Base of the logarithms used to calculate the entropy	`2`
`reference_score` ¶	`float \| Literal['mean'] \| Expr`	Baseline against which per-token contributions are interpreted. Defaults to 0. If `"mean"`, uses the Shannon entropy of the reference corpus as the reference score	`0`

Comparison.score.shannon_entropy