Comparison.exclude.by_frequency

Comparison.exclude.by_frequency ¶

by_frequency(
    less_than: float | None = None,
    greater_than: float | None = None,
    between: tuple[float, float] | None = None,
    inclusive: bool = True,
    corpus: EXCLUDE_CORPUS = "both",
) -> Comparison

Removes tokens whose frequency falls in the given range from the reference and comparison corpora.

Excluding by frequency narrows a comparison to the part of the frequency distribution that is of interest, such as dropping rare or common tokens. The range is tested against the frequencies the catalog was built from, so its bounds are counts. Tokens are excluded from both corpora, before they are scored, so the frequencies the scoring measure works with are normalized over the remaining vocabulary.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"the": 30, "good": 12, "bad": 8, "happy": 6, "sad": 4, "calm": 5},
        "after": {"the": 25, "good": 5, "bad": 7, "happy": 3, "sad": 9, "angry": 6},
    }
)

cl = cl.with_comparisons(
    wl.comp("before", "after").exclude.by_frequency(greater_than=20).score.proportion()
)

Exactly one of less_than, greater_than, and between gives the range to exclude. To exclude by multiple disjoint criteria, call the method multiple times with different parameters. A count means different things in corpora of different sizes, so use by_proportion to exclude by a share of each corpus instead.

Warning

Excluding tokens changes the normalized frequencies of every token that remains.

A token missing from one of the corpora has a frequency of zero, so corpus="either" paired with less_than removes every token that the two corpora do not share.

Parameters:

Name	Type	Description	Default
`less_than` ¶	`float \| None`	Exclude tokens with a frequency below this bound	`None`
`greater_than` ¶	`float \| None`	Exclude tokens with a frequency above this bound	`None`
`between` ¶	`tuple[float, float] \| None`	Exclude tokens with frequencies between the lower and upper bound	`None`
`inclusive` ¶	`bool`	Whether the bounds themselves are part of the range being excluded	`True`
`corpus` ¶	`EXCLUDE_CORPUS`	Which corpus's frequency the range is evaluated against. `"reference"` and `"comparison"` evaluate against a single corpus; `"both"` excludes a token only when the range holds in both corpora; `"either"` excludes a token when the range holds in either one	`'both'`

Returns:

Type	Description
`Comparison`	A new `Comparison` whose vocabulary excludes tokens with frequencies in the given range

Raises:

Type	Description
`ValueError`	If none of `less_than`, `greater_than`, and `between` is given, or if more than one of them is.
`ValueError`	If the lower bound of `between` is greater than its upper bound.
`ValueError`	If `corpus` is not a recognized corpus to test the range against.

Source

Comparison.exclude.by_frequency