Comparison.exclude.by_proportion

Comparison.exclude.by_proportion ¶

by_proportion(
    less_than: float | None = None,
    greater_than: float | None = None,
    between: tuple[float, float] | None = None,
    inclusive: bool = True,
    corpus: EXCLUDE_CORPUS = "both",
) -> Comparison

Removes tokens whose proportion of a corpus falls in the given range from the reference and comparison corpora.

Excluding by proportion narrows a comparison to the part of the frequency distribution that is of interest, using each token's share of its corpus rather than a count, so that the same bounds mean the same thing in corpora of different sizes. Tokens are excluded from both corpora, before they are scored, so the frequencies the scoring measure works with are normalized over the remaining vocabulary.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"the": 30, "good": 12, "bad": 8, "happy": 6, "sad": 4, "calm": 5},
        "after": {"the": 25, "good": 5, "bad": 7, "happy": 3, "sad": 9, "angry": 6},
    }
)

cl = cl.with_comparisons(
    wl.comp("before", "after").exclude.by_proportion(less_than=0.15).score.proportion()
)

Exactly one of less_than, greater_than, and between gives the range to exclude. To exclude by multiple disjoint criteria, call the method multiple times with different parameters. Proportions are taken over each corpus as the catalog gave it, before anything the comparison excludes, so the bounds mean the same thing no matter what else the comparison excludes or how it is scored.

Warning

Excluding tokens changes the normalized frequencies of every token that remains.

A token missing from one of the corpora has a proportion of zero, so corpus="either" paired with less_than removes every token that the two corpora do not share.

Parameters:

Name	Type	Description	Default
`less_than` ¶	`float \| None`	Exclude tokens making up less than this share of a corpus	`None`
`greater_than` ¶	`float \| None`	Exclude tokens making up more than this share of a corpus	`None`
`between` ¶	`tuple[float, float] \| None`	Exclude tokens whose share of a corpus is between the lower and upper bound	`None`
`inclusive` ¶	`bool`	Whether the bounds themselves are part of the range being excluded	`True`
`corpus` ¶	`EXCLUDE_CORPUS`	Which corpus's proportion the range is evaluated against. `"reference"` and `"comparison"` evaluate against a single corpus; `"both"` excludes a token only when the range holds in both corpora; `"either"` excludes a token when the range holds in either one	`'both'`

Returns:

Type	Description
`Comparison`	A new `Comparison` whose vocabulary excludes tokens with proportions in the given range

Raises:

Type	Description
`ValueError`	If none of `less_than`, `greater_than`, and `between` is given, or if more than one of them is.
`ValueError`	If the lower bound of `between` is greater than its upper bound.
`ValueError`	If `corpus` is not a recognized corpus to test the range against.
`ValueError`	If a bound is outside \([0, 1]\)

Source

Comparison.exclude.by_proportion