Comparison.exclude.tokens

Removes the given tokens from the the reference and comparison corpora.

Excluding tokens is a way of narrowing a comparison to the vocabulary that is of interest, such as removing stop words that carry little meaning of their own. Tokens are excluded from both corpora, before they are scored, so the frequencies the scoring measure works with are normalized over the remaining vocabulary.

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"the": 30, "good": 12, "bad": 8, "happy": 6, "sad": 4, "calm": 5},
        "after": {"the": 25, "good": 5, "bad": 7, "happy": 3, "sad": 9, "angry": 6},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").exclude.tokens(["the"]).score.proportion())

Scoring a comparison after excluding gives the same result as excluding after scoring. Excluding more than once removes the union of the tokens, and a token that is not in the vocabulary is ignored.

Parameters:

Returns:

Source