Skip to content

Comparison.normalize

Comparison.normalize 

normalize(by: NormalizeBy) -> Comparison

Returns a new Comparison whose token contributions are normalized onto a common scale.

Each scoring measure produces a per-token contribution \(\delta_\tau\). Normalization rescales every contribution by a single constant so that scores become interpretable and comparable across measures. There are two normalization strategies:

  • "sum_abs" divides by the sum of absolute contributions, \(\sum_\tau |\delta_\tau|\). The absolute values of the normalized contributions then sum to 1.
  • "abs_sum" divides by the absolute sum of contributions, \(|\sum_\tau \delta_\tau|\). The normalized contributions then sum to \(\pm 1\).

Normalizing again by the same strategy is a no-op; normalizing by a different strategy raises an error, since the contributions are already on a normalized scale.

Example 
import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").score.shannon_entropy().normalize("sum_abs"))
Warning

Using this method forces the comparison to be evaluated eagerly rather than lazily, because validating it requires computing the scores over the full vocabulary.

Parameters:

Name Type Description Default

by

NormalizeBy

The normalization strategy. "sum_abs" divides every token's contribution by \(\sum_\tau |\delta_\tau|\); "abs_sum" divides by \(|\sum_\tau \delta_\tau|\)

 required

Returns:

Type Description
Comparison

A new Comparison with normalized token contributions

Raises:

Type Description
ValueError

If the comparison is already normalized by a different strategy.
ValueError

If by is not a recognized normalization strategy.
ValueError

If by is "abs_sum" for a proportion comparison, whose contributions sum to zero by construction, so the absolute sum is always zero.

Source