Comparison.exclude.by_lexicon_score

Comparison.exclude.by_lexicon_score ¶

by_lexicon_score(
    less_than: float | None = None,
    greater_than: float | None = None,
    between: tuple[float, float] | None = None,
    inclusive: bool = True,
    corpus: ExcludeCorpus = "both",
) -> Comparison

Removes tokens whose lexicon score falls in the given range from the reference and comparison corpora.

The range is tested against the scores the comparison's lexicon assigns. Tokens are excluded from both corpora, before they are scored, so the frequencies the scoring measure works with are normalized over the remaining vocabulary.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"the": 30, "good": 12, "bad": 8, "happy": 6, "sad": 4, "calm": 5},
        "after": {"the": 25, "good": 5, "bad": 7, "happy": 3, "sad": 9, "angry": 6},
    }
)

cl = cl.with_comparisons(
    wl.comp("before", "after")
    .exclude.by_lexicon_score(between=(4.0, 6.0))
    .score.lexicon(wl.lex.labMT("english"))
)

Exactly one of less_than, greater_than, and between gives the range to exclude. To exclude by multiple disjoint criteria, call the method multiple times with different parameters.

Warning

Excluding tokens changes the normalized frequencies of every token that remains.

Parameters:

Name	Type	Description	Default
`less_than` ¶	`float \| None`	Exclude tokens with a lexicon score below this bound	`None`
`greater_than` ¶	`float \| None`	Exclude tokens with a lexicon score above this bound	`None`
`between` ¶	`tuple[float, float] \| None`	Exclude tokens with lexicon scores between the lower and upper bound	`None`
`inclusive` ¶	`bool`	Whether the bounds themselves are part of the range being excluded	`True`
`corpus` ¶	`ExcludeCorpus`	Which corpus's lexicon score the range is evaluated against. `"reference"` and `"comparison"` evaluate against the lexicons associated with each corpus; `"both"` excludes a token only when the range holds for the lexicons for both corpora; `"either"` excludes a token when the condition holds for either lexicon. All four specifications are equivalent when a single lexicon scores both corpora	`'both'`

Returns:

Type	Description
`Comparison`	A new `Comparison` whose vocabulary excludes tokens scoring in the given range

Raises:

Type	Description
`ValueError`	If none of `less_than`, `greater_than`, and `between` is given, or if more than one of them is.
`ValueError`	If the lower bound of `between` is greater than its upper bound.
`ValueError`	If `corpus` is not a recognized corpus to test the range against.
`ValueError`	If the comparison is not scored by a lexicon, so it has no lexicon scores to exclude by.
`ValueError`	If tokens scored by only one lexicon are imputed rather than borrowed or dropped, because an imputed score is a modeling choice rather than what a lexicon says about a token.

Source

Comparison.exclude.by_lexicon_score