Comparison.normalize

Comparison.normalize ¶

normalize(
    by: NormalizeBy, check_nonzero: bool = True
) -> Comparison

Returns a new Comparison whose token contributions are normalized onto a common scale.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").score.shannon_entropy().normalize("total_diff"))

Each scoring measure produces a per-token contribution \(\delta_\tau\). Normalization divides every contribution by a single constant, either the total difference between the corpora \(\sum_\tau |\delta_\tau|\) or their net difference \(|\sum_\tau \delta_\tau|\), so that scores become interpretable and comparable across measures.

For what each strategy means and how to read a normalized score, see Normalization.

Warning

By default, using this method forces the comparison to be evaluated eagerly rather than lazily, because validating it requires computing the scores over the full vocabulary. The eager validation can be skipped using the check_nonzero parameter.

Parameters:

Name	Type	Description	Default
`by` ¶	`NormalizeBy`	The normalization strategy. `"total_diff"` divides every token's contribution by the total difference \(\sum_\tau \|\delta_\tau\|\); `"net_diff"` divides by the net difference \(\|\sum_\tau \delta_\tau\|\)	required
`check_nonzero` ¶	`bool`	Whether to check whether the normalization constant is zero and raise an error. If `False`, the operation remains lazy, but the scores can become NaN when normalizing by `"total_diff"` or inf and -inf when normalizing by `"net_diff"` if the normalization constant is zero	`True`

Returns:

Type	Description
`Comparison`	A new `Comparison` with normalized token contributions

Raises:

Type	Description
`ValueError`	If `check_nonzero` is `True` and the normalization constant is zero. This is raised when the comparison is scored, not when `normalize` is called.
`ValueError`	If the comparison is already normalized by a different strategy.
`ValueError`	If `by` is not a recognized normalization strategy.
`ValueError`	If `by` is `"net_diff"` for a `proportion` comparison, whose contributions sum to zero by construction, so the net difference is always zero.

Source

Comparison.normalize