Catalog ¶

Catalog(
    data: DataFrame
    | LazyFrame
    | IntoFrame
    | Catalog
    | CatalogData
    | DictOfFreqs
    | ListOfFreqs
    | Dataset,
    token_col: str | None = None,
    corpora: Sequence[str] | None = None,
)

The workspace for corpora and making comparisons between them.

For specific construction functions, see the construction documentation.

Parameters:

Name	Type	Description	Default
`data`	`DataFrame \| LazyFrame \| IntoFrame \| Catalog \| CatalogData \| DictOfFreqs \| ListOfFreqs \| Dataset`	The corpora token-frequency data. Accepts a Polars DataFrame or LazyFrame, a dataframe from any other backend supported by Narwhals, `ListOfFreqs`, `DictOfFreqs`, or an existing `Catalog`. The class will attempt to infer the structure of the data and use the appropriate constructor.	required
`token_col`	`str \| None`	If the `data` is a dataframe, the column that contains the tokens (words, subwords, n-grams, etc.). Ignored otherwise	`None`
`corpora`	`Sequence[str] \| None`	If the `data` is a `ListOfFreqs`, the names of the corpora, listed in the same order as `data`. If not provided, will default to `corpus1`, `corpus2`, etc. If the `data` is a dataframe, the names of the columns containing token frequencies, which must be provided. Ignored otherwise	`None`

Raises:

Type	Description
`TypeError`	If `data` is not one of the supported types.

Source

Methods:

Name	Description
`from_catalog`	Creates a new catalog from an existing one.
`from_dataframe`	Creates a `Catalog` of corpora from any dataframe supported by Narwhals.
`from_dataset`	Creates a `Catalog` from a WordLevel pre-made dataset.
`from_dict_of_freqs`	Creates a `Catalog` of corpora from named token-to-frequency mappings.
`from_list_of_freqs`	Creates a `Catalog` of corpora from a sequence of token-to-frequency mappings, each
`from_polars`	Creates a `Catalog` of corpora from a Polars dataframe.
`with_comparisons`	Returns a new catalog with comparisons added to it.

Attributes:

Name	Type	Description
`comparisons`	`Mapping[str, ComparisonResult]`	Comparisons made between corpora in the catalog, keyed by each comparison's alias.
`df`	`DataFrame`	Dataframe representing the frequencies of tokens in the corpora in the catalog.
`plot`	`Plotter`	Namespace for plotting methods. See available plots.

Attributes¶

comparisons `property` ¶

comparisons: Mapping[str, ComparisonResult]

Comparisons made between corpora in the catalog, keyed by each comparison's alias.

Source

df `property` ¶

df: DataFrame

Dataframe representing the frequencies of tokens in the corpora in the catalog.

Source

plot `cached` `property` ¶

plot: Plotter

Namespace for plotting methods. See available plots.

Source

Methods:¶

from_catalog `classmethod` ¶

from_catalog(catalog: Catalog) -> Catalog

Creates a new catalog from an existing one.

The new catalog copies the data of the original one without mutating it.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

new_cl = wl.Catalog.from_catalog(cl)

Parameters:

Name	Type	Description	Default
`catalog`	`Catalog`	An existing `Catalog`	required

Returns:

Type	Description
`Catalog`	A new `Catalog` with the same internal data as the provided catalog

Source

from_dataframe `classmethod` ¶

from_dataframe(
    df: IntoFrame,
    token_col: str,
    corpora: Collection[str],
    normalize: bool = True,
) -> Catalog

Creates a Catalog of corpora from any dataframe supported by Narwhals.

The dataframe is converted to Polars before the catalog is constructed.

Warning

The catalog's engine is backed by Polars. Any dataframe it returns is a Polars dataframe, regardless of the backend that the data came from.

Polars dataframes are used as given, so a LazyFrame stays lazy. Dataframes from other lazy backends, like DuckDB and PySpark, are materialized in memory.

Example

import pandas as pd
import wordlevel as wl

df = pd.DataFrame(
    {
        "token": ["good", "bad", "happy", "sad"],
        "before": [12, 8, 6, 4],
        "after": [5, 7, 3, 9],
    }
)

cl = wl.Catalog.from_dataframe(df, token_col="token", corpora=["before", "after"])

Parameters:

Name	Type	Description	Default
`df`	`IntoFrame`	A dataframe containing corpus information, from any backend supported by Narwhals. Rows are tokens and columns are their frequencies in different corpora	required
`token_col`	`str`	The column that contains the tokens (words, subwords, n-grams, etc.)	required
`corpora`	`Collection[str]`	The columns that include the token frequencies for each corpus	required
`normalize`	`bool`	Whether to normalize the corpus frequencies for each corpus so that they are in the range \([0, 1]\) and sum to 1. The original corpus frequencies will be preserved	`True`

Returns:

Type	Description
`Catalog`	A `Catalog` of corpora representing the provided tokens and frequencies

Raises:

Type	Description
`TypeError`	If `df` is not a dataframe that can be converted to Polars.
`KeyError`	If `token_col` is not a column in `df`.
`KeyError`	If any column in `corpora` is not a column in `df`.
`ValueError`	If fewer than two columns are provided in `corpora`.

Source

from_dataset `classmethod` ¶

from_dataset(
    data: Dataset, corpora: Collection[str] | None = None
) -> Catalog

Creates a Catalog from a WordLevel pre-made dataset.

Parameters:

Name	Type	Description	Default
`data`	`Dataset`	A WordLevel dataset	required
`corpora`	`Collection[str] \| None`	The names of the corpora to include from the dataset	`None`

Returns:

Type	Description
`Catalog`	A `Catalog` of corpora representing the provided WordLevel dataset

Raises:

Type	Description
`KeyError`	If any corpus in `corpora` is not a valid specifier
`ValueError`	If fewer than two `corpora` are provided

Source

from_dict_of_freqs `classmethod` ¶

from_dict_of_freqs(freqs: DictOfFreqs) -> Catalog

Creates a Catalog of corpora from named token-to-frequency mappings.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

Parameters:

Name	Type	Description	Default
`freqs`	`DictOfFreqs`	A mapping of corpus names to token-to-frequency mappings, one per corpus. Each mapping is `{corpus: {token: frequency}}`, with one mapping per corpus	required

Returns:

Type	Description
`Catalog`	A `Catalog` of corpora representing the provided tokens and frequencies

Raises:

Type	Description
`ValueError`	If fewer than two corpora are provided in `freqs`.
`ValueError`	If any corpus in `freqs` has no token frequencies.

Source

from_list_of_freqs `classmethod` ¶

from_list_of_freqs(
    freqs: ListOfFreqs, corpora: Sequence[str] | None = None
) -> Catalog

Creates a Catalog of corpora from a sequence of token-to-frequency mappings, each representing the frequencies of tokens in a particular corpus.

Example

import wordlevel as wl

cl = wl.Catalog.from_list_of_freqs(
    [
        {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    ],
    corpora=["before", "after"],
)

Parameters:

Name	Type	Description	Default
`freqs`	`ListOfFreqs`	A sequence of token-to-frequency mappings, one per corpus. Each mapping is `{token: frequency}` for a single corpus	required
`corpora`	`Sequence[str] \| None`	The names of the corpora, listed in the same order as `freqs`. If not provided, will default to `corpus1`, `corpus2`, etc.	`None`

Returns:

Type	Description
`Catalog`	A `Catalog` of corpora representing the provided tokens and frequencies

Raises:

Type	Description
`ValueError`	If `freqs` and `corpora` have different lengths.
`ValueError`	If fewer than two corpora are provided in `freqs`.
`ValueError`	If any corpus in `freqs` has no token frequencies.

Source

from_polars `classmethod` ¶

from_polars(
    df: DataFrame | LazyFrame,
    token_col: str,
    corpora: Collection[str],
    normalize: bool = True,
) -> Catalog

Creates a Catalog of corpora from a Polars dataframe.

Example

import polars as pl
import wordlevel as wl

df = pl.DataFrame(
    {
        "token": ["good", "bad", "happy", "sad"],
        "before": [12, 8, 6, 4],
        "after": [5, 7, 3, 9],
    }
)

cl = wl.Catalog.from_polars(df, token_col="token", corpora=["before", "after"])

Parameters:

Name	Type	Description	Default
`df`	`DataFrame \| LazyFrame`	A Polars DataFrame or LazyFrame containing corpus information. Rows are tokens and columns are their frequencies in different corpora	required
`token_col`	`str`	The column that contains the tokens (words, subwords, n-grams, etc.)	required
`corpora`	`Collection[str]`	The columns that include the token frequencies for each corpus	required
`normalize`	`bool`	Whether to normalize the corpus frequencies for each corpus so that they are in the range \([0, 1]\) and sum to 1. The original corpus frequencies will be preserved	`True`

Returns:

Type	Description
`Catalog`	A `Catalog` of corpora representing the provided tokens and frequencies

Raises:

Type	Description
`KeyError`	If `token_col` is not a column in `df`.
`KeyError`	If any column in `corpora` is not a column in `df`.
`ValueError`	If fewer than two columns are provided in `corpora`.

Source

with_comparisons ¶

with_comparisons(*comparisons: Comparison) -> Catalog

Returns a new catalog with comparisons added to it.

New comparisons will overwrite existing comparisons with the same name.

Example

import wordlevel as wl

cl = wl.Catalog.from_dict_of_freqs(
    {
        "before": {"good": 12, "bad": 8, "happy": 6, "sad": 4},
        "after": {"good": 5, "bad": 7, "happy": 3, "sad": 9},
    }
)

cl = cl.with_comparisons(
    wl.comp("before", "after").score.kld().alias("kld_comparison"),
    wl.comp("before", "after").score.jsd().alias("jsd_comparison"),
)

Parameters:

Name	Type	Description	Default
`comparisons`	`Comparison`	The comparisons to add to the catalog	`()`

Returns:

Type	Description
`Catalog`	The catalog with the comparisons added to it

Source

Catalog ¶

Attributes¶

comparisons property ¶

df property ¶

plot cached property ¶

Methods:¶

from_catalog classmethod ¶

from_dataframe classmethod ¶

from_dataset classmethod ¶

from_dict_of_freqs classmethod ¶

from_list_of_freqs classmethod ¶

from_polars classmethod ¶

with_comparisons ¶

comparisons `property` ¶

df `property` ¶

plot `cached` `property` ¶

from_catalog `classmethod` ¶

from_dataframe `classmethod` ¶

from_dataset `classmethod` ¶

from_dict_of_freqs `classmethod` ¶

from_list_of_freqs `classmethod` ¶

from_polars `classmethod` ¶