LexiconLoader ¶

LexiconLoader(
    cache_dir: str | Path | None = None,
    hf_token: str | None = None,
    revision: Literal["cached", "latest"] | str = "cached",
)

An interface for (down)loading pre-defined lexicons that map tokens to scores.

wl.lex is a ready-made loader with the default options, and is all that most uses need. Construct a LexiconLoader to change where lexicons are cached, which HuggingFace token they are downloaded with, or which version of each lexicon is loaded.

Lexicons are downloaded from HuggingFace and cached locally. Each lexicon is versioned separately, so updating one does not change the others.

Example

import wordlevel as wl

latest_loader = wl.LexiconLoader(revision="latest")
labmt = latest_loader.labMT("english")

# The commit the lexicon was loaded at, to load exactly the same version elsewhere
pinned_loader = wl.LexiconLoader(revision=labmt.revision)
pinned_labmt = pinned_loader.labMT("english")

Parameters:

Name	Type	Description	Default
`cache_dir`	`str \| Path \| None`	The directory that downloaded lexicons are cached in. Defaults to the user's cache directory, which is shared by every project on the machine. Pass a directory within a project to keep its lexicons separate from other projects.	`None`
`hf_token`	`str \| None`	A HuggingFace access token to authenticate downloads with. If not given, the token is read from the `HF_TOKEN` environment variable or a local HuggingFace login, if either is available.	`None`
`revision`	`Literal['cached', 'latest'] \| str`	Which version of each lexicon to load. `"cached"` downloads from HuggingFace the first time that a lexicon is requested, and then only accesses that cached lexicon on later calls. `"latest"` checks HuggingFace for the latest version on every load, updating the cached copy if it has changed; if HuggingFace cannot be reached, the cached copy is loaded with a warning. Any other value is parsed as a commit hash, branch, or tag of the lexicon repository, and exactly that version is loaded, never a substitute. The version a lexicon was loaded at is recorded on its `revision`.	`'cached'`

Source

Methods:

Name	Description
`update_cached`	Updates every cached lexicon to its latest version on HuggingFace.

Methods:¶

update_cached ¶

update_cached() -> None

Updates every cached lexicon to its latest version on HuggingFace.

Only lexicons that have been updated on HuggingFace are downloaded again.

Example

import wordlevel as wl

wl.lex.update_cached()

Raises:

Type	Description
`LexiconConnectionError`	If HuggingFace could not be reached
`LexiconDownloadError`	If the lexicons could not be downloaded from HuggingFace

Source