Comparison.score.tsallis_entropy

Comparison.score.tsallis_entropy ¶

tsallis_entropy(
    alpha: float = 1,
    base: float = 2,
    reference_score: float | Literal["mean"] | Expr = 0,
) -> Comparison

Calculates the difference in Tsallis entropy between the reference and comparison corpora.

Tsallis entropy is a generalized measure of textual diversity, parameterized by an order \(\alpha\). When \(\alpha = 1\), the Tsallis entropy reduces to the Shannon entropy; when \(\alpha < 1\), more weight is given to rare tokens; and when \(\alpha > 1\), more weight is given to common tokens.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").score.tsallis_entropy())

When the reference_score is 0, the contribution \(\delta_\tau\) of a token \(\tau\) is:

\[ \delta_\tau = \frac{p_\tau^{(C)} - \left[ p_\tau^{(C)} \right]^\alpha}{\alpha - 1} - \frac{p_\tau^{(R)} - \left[ p_\tau^{(R)} \right]^\alpha}{\alpha - 1} \]

where \(p_\tau^{(R)}\) and \(p_\tau^{(C)}\) are the normalized frequencies of token \(\tau\) in \(R\) and \(C\) respectively, the reference and comparison corpora.

Each token is interpreted by whether it reinforces or offsets the corpus-level difference in entropies \(\Phi^{(C)} - \Phi^{(R)}\). A contribution with the same sign as the overall difference reinforces it, making its magnitude larger; a contribution with the opposite sign offsets it, pulling the difference back toward zero. For example, if \(\Phi^{(C)} - \Phi^{(R)} > 0\) (the comparison's entropy is higher), positive contributions reinforce the difference and negative contributions offset it; the interpretation flips when \(\Phi^{(C)} - \Phi^{(R)} < 0\).

When the reference_score is non-zero, the contribution of the token is:

\[ \delta_\tau = \left( p_\tau^{(C)} - p_\tau^{(R)} \right) \left[ \frac{1}{2} \left( \phi_\tau^{(C)} + \phi_\tau^{(R)} \right) - \Phi^{(\text{ref})} \right] + \frac{1}{2} \left( p_\tau^{(C)} + p_\tau^{(R)} \right) \left( \phi_\tau^{(C)} - \phi_\tau^{(R)} \right) \]

where \(\Phi^{(\text{ref})}\) is the user-provided reference score, and

\[ \phi_\tau^{(C)} = \frac{1 - \left[ p_\tau^{(C)} \right]^{\alpha - 1}}{\alpha - 1} \, , \, \phi_\tau^{(R)} = \frac{1 - \left[ p_\tau^{(R)} \right]^{\alpha - 1}}{\alpha - 1} \]

Reference

For more details, see the following paper:

Gallagher, R. J., Frank, M. R., Mitchell, L., Schwartz, A. J., Reagan, A. J., Danforth, C. M., & Dodds, P. S. (2021). Generalized word shift graphs: a method for visualizing and explaining pairwise comparisons between texts. EPJ Data Science, 10(1), 4.

See Also

Score	Description
`Comparison.score.shannon_entropy`	Scores corpora in the special case when \(\alpha=1\)

Parameters:

Name	Type	Description	Default
`alpha` ¶	`float`	Order of the Tsallis entropy. \(\alpha = 1\) recovers Shannon entropy. Larger values place more weight on common tokens, and smaller values place more weight on rare tokens	`1`
`base` ¶	`float`	Base of the logarithms used to calculate the entropy when \(\alpha = 1\). Ignored otherwise	`2`
`reference_score` ¶	`float \| Literal['mean'] \| Expr`	Baseline against which per-token contributions are interpreted. Defaults to 0. If `"mean"`, uses the Tsallis entropy of the reference corpus as the reference score	`0`

Returns:

Type	Description
`Comparison`	A new `Comparison` scored by the difference in Tsallis entropies

Source

Comparison.score.tsallis_entropy