Skip to content

Comparison.score.rank_divergence

Comparison.score.rank_divergence 

rank_divergence(alpha: float = 1.0) -> Comparison

Calculates the rank-turbulence divergence between the reference and comparison corpora.

Example 
import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").score.rank_divergence())

Rank-turbulence divergence compares corpora by the ranks of their tokens when ordered by relative frequency. When \(\alpha > 0\), the contribution \(\delta_\tau\) of a token \(\tau\) is:

\[ \delta_\tau = \frac{1}{\mathcal{N}_\alpha} \, \left| \frac{1}{\left[ r_\tau^{(R)} \right]^\alpha} - \frac{1}{\left[ r_\tau^{(C)} \right]^\alpha} \right| ^{ 1 / (\alpha + 1)} \]

where \(r_\tau^{(R)}\) and \(r_\tau^{(C)}\) are the ranks of \(\tau\) in the reference corpus \(R\) and the comparison corpus \(C\), and \(\mathcal{N}_\alpha\) is a normalization constant based on \(\alpha\). When \(\alpha = 0\), the contribution is instead \(\left| \ln \left( r_\tau^{(R)} / r_\tau^{(C)} \right) \right| / \mathcal{N}_0\).

For more details, including the effect of tuning \(\alpha\), see Divergences.

Reference

For details, see the following paper:

Dodds, P. S., Minot, J. R., Arnold, M. V., Alshaabi, T., Adams, J. L., Dewhurst, D. R., Gray, T. J., Frank, M. R., Reagan, A. J., & Danforth, C. M. (2023). Allotaxonometry and rank-turbulence divergence: A universal instrument for comparing complex systems. EPJ Data Science, 12(1), 37.

Parameters:

Name Type Description Default

alpha

float

Order of the rank divergence. Larger values place more weight on common tokens, and smaller values place more weight on rare tokens

 1.0

Returns:

Type Description
Comparison

A new Comparison scored by the rank-turbulence divergence

Source