Skip to content

Comparison.score.rank_divergence

Comparison.score.rank_divergence 

rank_divergence(alpha: float = 1.0) -> Comparison

Calculates the rank-turbulence divergence between the reference and comparison corpora.

Rank-turbulence divergence compares corpora by comparing the ranks of tokens in each corpus when ordered by relative frequency. The measure is symmetric and bounded between 0 and 1, where 0 indicates that the corpora are identical, and 1 indicates that they are maximally divergent. It is also parameterized by an order parameter \(\alpha\). When \(\alpha < 1\), more weight is given to rare tokens; when \(\alpha > 1\), more weight is given to common tokens.

Example 
import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(wl.comp("before", "after").score.rank_divergence())

When \(\alpha > 0\), the contribution \(\delta_\tau\) of a token \(\tau\) is:

\[ \delta_\tau = \frac{1}{\mathcal{N}_\alpha} \, \left| \frac{1}{\left[ r_\tau^{(R)} \right]^\alpha} - \frac{1}{\left[ r_\tau^{(C)} \right]^\alpha} \right| ^{ 1 / (\alpha + 1)} \]

where \(r_\tau^{(R)}\) and \(r_\tau^{(C)}\) are the ranks of token \(\tau\) in \(R\) and \(C\) respectively, the reference and comparison corpora, and \(\mathcal{N}_\alpha\) is a normalization constant based on \(\alpha\).

When \(\alpha = 0\), the contribution of the token is:

\[ \delta_\tau = \frac{1}{N_0} \, \left| \ln \frac{r_\tau^{(R)}}{r_\tau^{(C)}} \right| \]
Reference

For details, see the following paper:

Dodds, P. S., Minot, J. R., Arnold, M. V., Alshaabi, T., Adams, J. L., Dewhurst, D. R., Gray, T. J., Frank, M. R., Reagan, A. J., & Danforth, C. M. (2023). Allotaxonometry and rank-turbulence divergence: A universal instrument for comparing complex systems. EPJ Data Science, 12(1), 37.

Parameters:

Name Type Description Default

alpha

float

Order of the rank divergence. Larger values place more weight on common tokens, and smaller values place more weight on rare tokens

 1.0

Returns:

Type Description
Comparison

A new Comparison scored by the rank-turbulence divergence

Source