Comparison.score.proportion

Comparison.score.proportion ¶

proportion() -> Comparison

Calculates the proportion difference between the reference and comparison corpora.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").score.proportion())

Each token's contribution is the difference in its normalized frequency (i.e. proportion) within each corpus. The contribution \(\delta_\tau\) of a token \(\tau\) is:

\[ \delta_\tau = p_\tau^{(C)} - p_\tau^{(R)} \]

where \(p_\tau^{(R)}\) and \(p_\tau^{(C)}\) are the normalized frequencies of token \(\tau\) in \(R\) and \(C\) respectively, the reference and comparison corpora. Positive contributions identify tokens that occur relatively more often in the comparison; negative contributions identify tokens that occur relatively more often in the reference.

Reference

For more details, see the following paper:

Gallagher, R. J., Frank, M. R., Mitchell, L., Schwartz, A. J., Reagan, A. J., Danforth, C. M., & Dodds, P. S. (2021). Generalized word shift graphs: a method for visualizing and explaining pairwise comparisons between texts. EPJ Data Science, 10(1), 4.

Score	Description
`Comparison.score.frequency`	Scores corpora by raw frequencies
`Comparison.score.shannon_entropy`	Scores corpora by normalized frequencies weighted by surprisals