Comparison.score.lexicon

Comparison.score.lexicon ¶

lexicon(
    lexicon: Lexicon
    | Mapping[str, float]
    | DataFrame
    | LazyFrame
    | None = None,
    lexicon_reference: Lexicon
    | Mapping[str, float]
    | DataFrame
    | LazyFrame
    | None = None,
    lexicon_comparison: Lexicon
    | Mapping[str, float]
    | DataFrame
    | LazyFrame
    | None = None,
    lexicon_token_col: str | None = None,
    lexicon_score_col: str | None = None,
    reference_score: float
    | Literal["mean", "center"]
    | Expr = 0,
    impute: float
    | Literal["borrow", "mean"]
    | Expr = "borrow",
) -> Comparison

Calculates the difference in lexicon-weighted averages between the reference and comparison corpora.

The supplied lexicon associates a score with each token. The user can provide a single lexicon to use for both corpora, or one lexicon for each. The overall score for each corpus individually is the lexicon-weighted average. Specifically, if \(p_\tau\) is the normalized frequency of a token \(\tau\) in a corpus, and \(\phi_\tau\) is the score assigned to it by the lexicon, then the corpus-level score is \(\Phi = \sum_\tau p_\tau \phi_\tau\). This method takes the difference in weighted averages between the reference and comparison corpora, \(\Phi^{(C)} - \Phi^{(R)}\).

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").score.lexicon(wl.lex.labMT("english")))

When the reference_score is 0, the contribution \(\delta_\tau\) of a token \(\tau\) is:

\[ \delta_\tau = \phi^{(C)}_\tau p_\tau^{(C)} - \phi^{(R)}_\tau p_\tau^{(R)} \]

where \(\phi_\tau^{(C)}\) is the score of token \(\tau\) in the comparison corpus, and \(p_\tau^{(C)}\) is its normalized frequency. Similarly, \(\phi_\tau^{(R)}\) and \(p_\tau^{(R)}\) are the score and normalized frequency of \(\tau\) in the reference corpus.

Each token is interpreted by whether it reinforces or offsets the corpus-level difference in weighted averages \(\Phi^{(C)} - \Phi^{(R)}\). A contribution with the same sign as the overall difference reinforces it, making its magnitude larger; a contribution with the opposite sign offsets it, pulling the difference back toward zero. For example, if \(\Phi^{(C)} - \Phi^{(R)} > 0\) (the comparison's weighted average is higher), positive contributions reinforce the difference and negative contributions offset it; the interpretation flips when \(\Phi^{(C)} - \Phi^{(R)} < 0\).

If the reference_score is non-zero, and the score of a word is the same in both corpora, i.e. \(\phi_\tau = \phi^{(C)}_\tau = \phi^{(R)}_\tau\), then a token's contribution is:

\[ \delta_\tau = \left( p_\tau^{(C)} - p_\tau^{(R)} \right) \left( \phi_\tau - \Phi^{(\text{ref})} \right) \]

where \(\Phi^{(\text{ref})}\) is the user-provided reference score. If the reference_score is non-zero, and the score of a word is corpus dependent, then a token's contribution is:

\[ \delta_\tau = \left( p_\tau^{(C)} - p_\tau^{(R)} \right) \left[ \frac{1}{2} \left( \phi_\tau^{(C)} + \phi_\tau^{(R)} \right) - \Phi^{(\text{ref})} \right] + \frac{1}{2} \left( p_\tau^{(C)} + p_\tau^{(R)} \right) \left( \phi_\tau^{(C)} - \phi_\tau^{(R)} \right) \]

Reference

For more details, see the following paper:

Gallagher, R. J., Frank, M. R., Mitchell, L., Schwartz, A. J., Reagan, A. J., Danforth, C. M., & Dodds, P. S. (2021). Generalized word shift graphs: a method for visualizing and explaining pairwise comparisons between texts. EPJ Data Science, 10(1), 4.

Parameters:

Name	Type	Description	Default
`lexicon` ¶	`Lexicon \| Mapping[str, float] \| DataFrame \| LazyFrame \| None`	Lexicon associating tokens with scores. Accepts a mapping `{token: score}` or a Polars dataframe with columns indicating the tokens and scores. Also accepts a pre-defined lexicon	`None`
`lexicon_reference` ¶	`Lexicon \| Mapping[str, float] \| DataFrame \| LazyFrame \| None`	Lexicon associating tokens with scores for the reference corpus	`None`
`lexicon_comparison` ¶	`Lexicon \| Mapping[str, float] \| DataFrame \| LazyFrame \| None`	Lexicon associating tokens with scores for the comparison corpus	`None`
`lexicon_token_col` ¶	`str \| None`	Name of the token column when a dataframe is supplied for the lexicon	`None`
`lexicon_score_col` ¶	`str \| None`	Name of the score column when a dataframe is supplied for the lexicon	`None`
`reference_score` ¶	`float \| Literal['mean', 'center'] \| Expr`	Baseline against which per-token contributions are interpreted. Defaults to the "center" of a lexicon's numerical scale when a `Lexicon` is supplied; otherwise it defaults to 0. If `"mean"`, uses the weighted average of the reference corpus as the reference score	`0`
`impute` ¶	`float \| Literal['borrow', 'mean'] \| Expr`	How to score tokens when a score exists in one lexicon for the reference or comparison but not the other. `"borrow"` applies the score from the other corpus. `"mean"` uses the mean score of the corpus's lexicon (not weighted by frequency). A float or Polars expression can also be passed to use a specific value as the imputed score	`'borrow'`

Returns:

Type	Description
`Comparison`	A new `Comparison` scored by the difference in lexicon-weighted averages

Raises:

Type	Description
`ValueError`	If no lexicon is supplied (`lexicon`, `lexicon_reference`, and `lexicon_comparison` are all `None`).
`ValueError`	If `lexicon` is supplied alongside `lexicon_reference` or `lexicon_comparison` (the single-lexicon and per-corpus-lexicon paths are mutually exclusive).

Source

Comparison.score.lexicon