Comparison.exclude.by_rank

Comparison.exclude.by_rank ¶

by_rank(
    within_rank: float | None = None,
    beyond_rank: float | None = None,
    between: tuple[float, float] | None = None,
    inclusive: bool = True,
    corpus: ExcludeCorpus = "both",
    tie_method: RankTies = "average",
) -> Comparison

Removes tokens whose frequency rank falls in the given range from the reference and comparison corpora.

The range is tested against each token's frequency rank within its corpus, where rank 1 is the most common token, so that its bounds are vocabulary sizes rather than frequencies. Tokens are excluded from both corpora, before they are scored, so the frequencies the scoring measure works with are normalized over the remaining vocabulary.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"the": 30, "good": 12, "bad": 8, "happy": 6, "sad": 4, "calm": 5},
        "after": {"the": 25, "good": 5, "bad": 7, "happy": 3, "sad": 9, "angry": 6},
    }
)

cl = cl.with_comparisons(
    wl.comp("before", "after").exclude.by_rank(within_rank=2).score.proportion()
)

Exactly one of within_rank, beyond_rank, and between gives the range to exclude. To exclude by multiple disjoint criteria, call the method multiple times with different parameters. Ranks are taken over each corpus as the catalog gave it, before anything the comparison excludes, so the bounds mean the same thing no matter what else the comparison excludes or how it is scored.

Warning

Excluding tokens changes the normalized frequencies of every token that remains.

Tokens of equal frequency share a rank, and a tie block is excluded whole when that shared rank falls in the range. A bound is therefore neither a floor nor a ceiling on the size of the surviving vocabulary, especially in the tail, where the tokens a corpus uses once form a single large block.

A token missing from one of the corpora has a frequency of zero there, placing it in that corpus's bottom tie block, so corpus="either" paired with beyond_rank removes every token that the two corpora do not share.

Parameters:

Name	Type	Description	Default
`within_rank` ¶	`float \| None`	Exclude the tokens ranked within this bound. For example, `within_rank=100` excludes the 100 most frequently used tokens of a corpus	`None`
`beyond_rank` ¶	`float \| None`	Exclude the tokens ranked beyond this bound. For example, `beyond_rank=1000` excludes the tokens of a corpus that are not within its 1000 most frequently used	`None`
`between` ¶	`tuple[float, float] \| None`	Exclude tokens whose rank is between the lower and upper bound	`None`
`inclusive` ¶	`bool`	Whether the bounds themselves are part of the range being excluded	`True`
`corpus` ¶	`ExcludeCorpus`	Which corpus's rank the range is evaluated against. `"reference"` and `"comparison"` evaluate against a single corpus; `"both"` excludes a token only when the range holds in both corpora; `"either"` excludes a token when the range holds in either one	`'both'`
`tie_method` ¶	`RankTies`	How tokens of equal frequency are ranked	`'average'`

Returns:

Type	Description
`Comparison`	A new `Comparison` whose vocabulary excludes tokens with ranks in the given range

Raises:

Type	Description
`ValueError`	If none of `within_rank`, `beyond_rank`, and `between` is given, or if more than one of them is.
`ValueError`	If the lower bound of `between` is greater than its upper bound.
`ValueError`	If `corpus` is not a recognized corpus to test the range against.
`ValueError`	If a bound is less than one, since rank 1 is the most common token of a corpus.
`ValueError`	If `tie_method` is not a recognized way of ranking tokens of equal frequency.

Source

Comparison.exclude.by_rank