Comparison.score.lexicon

Comparison.score.lexicon ¶

lexicon(
    lexicon: Lexicon
    | Mapping[str, float]
    | IntoFrame
    | None = None,
    lexicon_reference: Lexicon
    | Mapping[str, float]
    | IntoFrame
    | None = None,
    lexicon_comparison: Lexicon
    | Mapping[str, float]
    | IntoFrame
    | None = None,
    lexicon_token_col: str | None = None,
    lexicon_score_col: str | None = None,
    reference_score: float
    | Literal["mean", "center"]
    | Expr = 0,
    scored_in_one_lexicon: float
    | ScoredInOneLexicon
    | Expr = "borrow",
) -> Comparison

Calculates the difference in lexicon-weighted averages between the reference and comparison corpora.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").score.lexicon(wl.lex.labMT("english")))

The supplied lexicon associates a score with each token. The user can provide a single lexicon to use for both corpora, or one lexicon for each. Each corpus is summarized by its lexicon-weighted average \(\Phi = \sum_\tau p_\tau \phi_\tau\), and this method takes the difference between the weighted averages of the reference corpus \(R\) and the comparison corpus \(C\):

\[ \Phi^{(C)} - \Phi^{(R)} = \sum_\tau p_\tau^{(C)} \phi_\tau^{(C)} - \sum_\tau p_\tau^{(R)} \phi_\tau^{(R)} \]

where \(p_\tau^{(C)}\) is the normalized frequency of a token \(\tau\) in \(C\) and \(\phi_\tau^{(C)}\) is the score the lexicon assigns it, and likewise for \(R\).

For how each token contributes to that difference, and how the reference_score shapes the contributions, see more on word shift graphs.

Warning

A lexicon dataframe is converted to Polars when this method is called. Polars dataframes are used as given, so a LazyFrame stays lazy. Dataframes from other lazy backends, like DuckDB and PySpark, are materialized in memory.

Reference

For more details, see the following paper:

Gallagher, R. J., Frank, M. R., Mitchell, L., Schwartz, A. J., Reagan, A. J., Danforth, C. M., & Dodds, P. S. (2021). Generalized word shift graphs: a method for visualizing and explaining pairwise comparisons between texts. EPJ Data Science, 10(1), 4.

Parameters:

Name	Type	Description	Default
`lexicon` ¶	`Lexicon \| Mapping[str, float] \| IntoFrame \| None`	Lexicon associating tokens with scores. Accepts a mapping `{token: score}` or a dataframe with columns indicating the tokens and scores, from any backend supported by Narwhals. Also accepts a pre-defined lexicon	`None`
`lexicon_reference` ¶	`Lexicon \| Mapping[str, float] \| IntoFrame \| None`	Lexicon associating tokens with scores for the reference corpus. Accepts the same forms as `lexicon`	`None`
`lexicon_comparison` ¶	`Lexicon \| Mapping[str, float] \| IntoFrame \| None`	Lexicon associating tokens with scores for the comparison corpus. Accepts the same forms as `lexicon`	`None`
`lexicon_token_col` ¶	`str \| None`	Name of the token column when a dataframe is supplied for the lexicon	`None`
`lexicon_score_col` ¶	`str \| None`	Name of the score column when a dataframe is supplied for the lexicon	`None`
`reference_score` ¶	`float \| Literal['mean', 'center'] \| Expr`	Baseline against which per-token contributions are interpreted. Defaults to 0. If "center' and a WordLevel lexicon is provided, uses the center of that lexicon's numerical scale. If `"mean"`, uses the weighted average of the reference corpus as the reference score	`0`
`scored_in_one_lexicon` ¶	`float \| ScoredInOneLexicon \| Expr`	How to handle a token that is scored by one of `lexicon_reference` or `lexicon_comparison` but not the other. Only applies when the two lexicons differ. `"borrow"` applies the score from the other corpus's lexicon, so the token contributes only through its change in frequency. `"drop"` removes the token from the comparison. `"mean"` imputes the mean score of the corpus's own lexicon (not weighted by frequency), and a float or Polars expression imputes that value instead. Because each corpus imputes from its own lexicon, an imputed token can have a different score in each corpus. Tokens that are in neither lexicon are always dropped	`'borrow'`

Returns:

Type	Description
`Comparison`	A new `Comparison` scored by the difference in lexicon-weighted averages

Raises:

Type	Description
`ValueError`	If no lexicon is supplied (`lexicon`, `lexicon_reference`, and `lexicon_comparison` are all `None`).
`ValueError`	If `lexicon` is supplied alongside `lexicon_reference` or `lexicon_comparison` (the single-lexicon and per-corpus-lexicon paths are mutually exclusive).
`ValueError`	If `scored_in_one_lexicon` is a string other than `"borrow"`, `"drop"`, or `"mean"`.

Source

Comparison.score.lexicon