Text Modules

taters.text.analyze_with_archetypes

analyze_with_archetypes

analyze_with_archetypes(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    on_progress=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    archetype_csvs,
    encoding="utf-8-sig",
    delimiter=",",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    model_name="sentence-transformers/all-roberta-large-v1",
    device="auto",
    mean_center_vectors=True,
    fisher_z_transform=False,
    rounding=4
)

Compute archetype scores for text rows and write a wide, analysis-ready features CSV.

This function supports three input modes:

1. analysis_csv — Use a prebuilt CSV with exactly two columns: text_id and text.
2. csv_path — Gather text from an arbitrary CSV by specifying text_cols (and optionally id_cols and group_by) to construct an analysis-ready CSV on the fly.
3. txt_dir — Gather text from a folder of .txt files.

Archetype scoring is delegated to a middle layer that embeds text with a Sentence-Transformers model and evaluates cosine similarity to one or more archetype CSVs. If out_features_csv is omitted, the default path is ./features/archetypes/<analysis_ready_filename>.

Parameters:

Name	Type	Description	Default
csv_path	str or Path	Source CSV for gathering. Mutually exclusive with txt_dir and analysis_csv.	None
txt_dir	str or Path	Folder of .txt files to gather from. Mutually exclusive with the other input modes.	None
analysis_csv	str or Path	Precomputed analysis-ready CSV containing exactly the columns text_id and text.	None
gathered_csv	str or Path	Where to write the intermediate "analysis-ready" table built from csv_path or txt_dir. By default it lands beside the source -- which means analyzing a spreadsheet in someone's Downloads folder writes a file into their Downloads folder. Pass this to keep the intermediate with the rest of a run's output instead. Ignored when analysis_csv is given, because then no gathering happens.	None
on_progress	callable	Called as on_progress(done, total, message=None) so a UI can show a real bar instead of a spinner. Injected automatically by the pipeline runner for any step function that declares this parameter. See :mod:taters.helpers.progress for the contract.	None
out_features_csv	str or Path	Output path for the features CSV. If None, defaults to ./features/archetypes/<analysis_ready_filename>.	None
overwrite_existing	bool	If False and the output file already exists, skip recomputation and return the existing path. This also controls the intermediate analysis-ready CSV: when True, it is rebuilt from the current source instead of reusing a stale copy from an earlier run.	False
archetype_csvs	Sequence[str or Path]	One or more archetype CSVs (name → seed phrases). Directories are allowed and expanded recursively to all .csv files.	required
encoding	str	Text encoding for CSV I/O.	"utf-8-sig"
delimiter	str	Field delimiter for CSV I/O.	","
text_cols	Sequence[str]	When gathering from a CSV: column(s) that contain text. Used only if csv_path is provided.	("text",)
id_cols	Sequence[str]	When gathering from a CSV: optional ID columns to carry into grouping (e.g., ["speaker"]).	None
mode	(concat, separate)	Gathering behavior when multiple text_cols are provided. "concat" joins into a single text field; "separate" creates one row per text column.	"concat"
group_by	Sequence[str]	Optional grouping keys used during gathering (e.g., ["speaker"]). In "concat" mode, members are concatenated into one row per group.	None
joiner	str	Separator used when concatenating multiple text chunks.	" "
num_buckets	int	Number of temporary hash buckets used for scalable CSV gathering.	512
max_open_bucket_files	int	Maximum number of bucket files to keep open concurrently during gathering.	64
tmp_root	str or Path	Root directory for temporary files used by gathering.	None
recursive	bool	When gathering from a text folder, whether to recurse into subdirectories.	True
pattern	str	Filename glob used when gathering from a text folder.	"*.txt"
id_from	(stem, name, path)	How to derive the text_id when gathering from a text folder.	"stem"
include_source_path	bool	Whether to include the absolute source path as an additional column when gathering from a text folder.	True
device	(auto, cuda, cpu)	Where to run the embedding model. "auto" uses the GPU when torch reports one that works and falls back to the CPU when it does not; "cuda" insists and raises if it cannot; "cpu" never touches the GPU.	"auto"
model_name	str	Sentence-Transformers model used to embed text for archetype scoring.	"sentence-transformers/all-roberta-large-v1"
mean_center_vectors	bool	If True, mean-center embedding vectors prior to scoring.	True
fisher_z_transform	bool	If True, apply the Fisher z-transform to correlations.	False
workers	int	Parallel processes for reading documents. 0 means automatic: three-quarters of the logical cores; 1 turns parallelism off. Output files are identical whatever the worker count.	0
rounding	int	Number of decimal places to round numeric outputs. Use None to disable rounding.	4

Returns:

Type	Description
Path	Path to the written features CSV.

Raises:

Type	Description
FileNotFoundError	If an input file or folder does not exist, or an archetype CSV path is invalid.
ValueError	If required arguments are incompatible or missing (e.g., no input mode chosen), or if the analysis-ready CSV lacks text_id/text columns.

Examples:

Run on a transcript CSV, grouped by speaker:

>>> analyze_with_archetypes(
...     csv_path="transcripts/session.csv",
...     text_cols=["text"],
...     id_cols=["speaker"],
...     group_by=["speaker"],
...     archetype_csvs=["dictionaries/archetypes"],
...     model_name="sentence-transformers/all-roberta-large-v1",
... )
PosixPath('.../features/archetypes/session.csv')

Notes

If out_features_csv exists and overwrite_existing=False, the existing path is returned without recomputation. Directories passed in archetype_csvs are expanded recursively to all .csv files and deduplicated before scoring.

Source code in src\taters\text\analyze_with_archetypes.py

@records_settings(binding=TEXT_INPUT, grain=TEXT_GRAIN,
                  outputs=("out_features_csv",),
                  bookkeeping=("WC",),
                  assets={"archetype_csvs": "archetypes"})
def analyze_with_archetypes(
    *,
    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[Union[str, Path]] = None,
    txt_dir: Optional[Union[str, Path]] = None,
    analysis_csv: Optional[Union[str, Path]] = None,   # if given, we skip gathering
    gathered_csv: Optional[Union[str, Path]] = None,
    on_progress: Optional[Callable[[int, int], None]] = None,

    # ----- Output -----
    out_features_csv: Optional[Union[str, Path]] = None,
    overwrite_existing: bool = False,
    workers: int = 0,  # if the file already exists, let's not overwrite by default

    # ----- Archetype CSVs (one or more) -----
    archetype_csvs: Sequence[Union[str, Path]],

    # ====== SHARED I/O OPTIONS ======
    encoding: str = "utf-8-sig",
    delimiter: str = ",",

    # ====== CSV GATHER OPTIONS (when csv_path is provided) ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[Union[str, Path]] = None,

    # ====== TXT FOLDER GATHER OPTIONS (when txt_dir is provided) ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ====== Archetyper scoring options ======
    model_name: str = "sentence-transformers/all-roberta-large-v1",
    device: Optional[str] = "auto",
    mean_center_vectors: bool = True,
    fisher_z_transform: bool = False,
    rounding: int = 4,
) -> Path:
    """
    Compute archetype scores for text rows and write a wide, analysis-ready features CSV.

    This function supports three input modes:

    1. ``analysis_csv`` — Use a prebuilt CSV with exactly two columns: ``text_id`` and ``text``.
    2. ``csv_path`` — Gather text from an arbitrary CSV by specifying ``text_cols`` (and optionally
    ``id_cols`` and ``group_by``) to construct an analysis-ready CSV on the fly.
    3. ``txt_dir`` — Gather text from a folder of ``.txt`` files.

    Archetype scoring is delegated to a middle layer that embeds text with a Sentence-Transformers
    model and evaluates cosine similarity to one or more archetype CSVs. If ``out_features_csv`` is
    omitted, the default path is ``./features/archetypes/<analysis_ready_filename>``.

    Parameters
    ----------
    csv_path : str or pathlib.Path, optional
        Source CSV for gathering. Mutually exclusive with ``txt_dir`` and ``analysis_csv``.
    txt_dir : str or pathlib.Path, optional
        Folder of ``.txt`` files to gather from. Mutually exclusive with the other input modes.
    analysis_csv : str or pathlib.Path, optional
        Precomputed analysis-ready CSV containing exactly the columns ``text_id`` and ``text``.
    gathered_csv : str or pathlib.Path, optional
        Where to write the intermediate "analysis-ready" table built from
        ``csv_path`` or ``txt_dir``.

        By default it lands beside the *source* -- which means analyzing a
        spreadsheet in someone's Downloads folder writes a file into their
        Downloads folder. Pass this to keep the intermediate with the rest of a
        run's output instead. Ignored when ``analysis_csv`` is given, because
        then no gathering happens.
    on_progress : callable, optional
        Called as ``on_progress(done, total, message=None)`` so a UI can show a
        real bar instead of a spinner. Injected automatically by the pipeline
        runner for any step function that declares this parameter. See
        :mod:`taters.helpers.progress` for the contract.
    out_features_csv : str or pathlib.Path, optional
        Output path for the features CSV. If ``None``, defaults to
        ``./features/archetypes/<analysis_ready_filename>``.
    overwrite_existing : bool, default=False
        If ``False`` and the output file already exists, skip recomputation and return the existing path.
        This also controls the intermediate analysis-ready CSV: when ``True``, it is rebuilt
        from the current source instead of reusing a stale copy from an earlier run.
    archetype_csvs : Sequence[str or pathlib.Path]
        One or more archetype CSVs (name → seed phrases). Directories are allowed and expanded
        recursively to all ``.csv`` files.
    encoding : str, default="utf-8-sig"
        Text encoding for CSV I/O.
    delimiter : str, default=","
        Field delimiter for CSV I/O.
    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV: column(s) that contain text. Used only if ``csv_path`` is provided.
    id_cols : Sequence[str], optional
        When gathering from a CSV: optional ID columns to carry into grouping (e.g., ``["speaker"]``).
    mode : {"concat", "separate"}, default="concat"
        Gathering behavior when multiple ``text_cols`` are provided. ``"concat"`` joins into a single
        text field; ``"separate"`` creates one row per text column.
    group_by : Sequence[str], optional
        Optional grouping keys used during gathering (e.g., ``["speaker"]``). In ``"concat"`` mode,
        members are concatenated into one row per group.
    joiner : str, default=" "
        Separator used when concatenating multiple text chunks.
    num_buckets : int, default=512
        Number of temporary hash buckets used for scalable CSV gathering.
    max_open_bucket_files : int, default=64
        Maximum number of bucket files to keep open concurrently during gathering.
    tmp_root : str or pathlib.Path, optional
        Root directory for temporary files used by gathering.
    recursive : bool, default=True
        When gathering from a text folder, whether to recurse into subdirectories.
    pattern : str, default="*.txt"
        Filename glob used when gathering from a text folder.
    id_from : {"stem", "name", "path"}, default="stem"
        How to derive the ``text_id`` when gathering from a text folder.
    include_source_path : bool, default=True
        Whether to include the absolute source path as an additional column when gathering from a text folder.
    device : {"auto", "cuda", "cpu"} | None, default "auto"
        Where to run the embedding model. "auto" uses the GPU when torch reports
        one that works and falls back to the CPU when it does not; "cuda"
        insists and raises if it cannot; "cpu" never touches the GPU.
    model_name : str, default="sentence-transformers/all-roberta-large-v1"
        Sentence-Transformers model used to embed text for archetype scoring.
    mean_center_vectors : bool, default=True
        If ``True``, mean-center embedding vectors prior to scoring.
    fisher_z_transform : bool, default=False
        If ``True``, apply the Fisher z-transform to correlations.
    workers : int, default=0
        Parallel processes for reading documents. ``0`` means automatic:
        three-quarters of the logical cores; ``1`` turns parallelism off. Output files are
        identical whatever the worker count.
    rounding : int, default=4
        Number of decimal places to round numeric outputs. Use ``None`` to disable rounding.

    Returns
    -------
    pathlib.Path
        Path to the written features CSV.

    Raises
    ------
    FileNotFoundError
        If an input file or folder does not exist, or an archetype CSV path is invalid.
    ValueError
        If required arguments are incompatible or missing (e.g., no input mode chosen),
        or if the analysis-ready CSV lacks ``text_id``/``text`` columns.

    Examples
    --------
    Run on a transcript CSV, grouped by speaker:

    >>> analyze_with_archetypes(
    ...     csv_path="transcripts/session.csv",
    ...     text_cols=["text"],
    ...     id_cols=["speaker"],
    ...     group_by=["speaker"],
    ...     archetype_csvs=["dictionaries/archetypes"],
    ...     model_name="sentence-transformers/all-roberta-large-v1",
    ... )
    PosixPath('.../features/archetypes/session.csv')

    Notes
    -----
    If ``out_features_csv`` exists and ``overwrite_existing=False``, the existing path is returned
    without recomputation. Directories passed in ``archetype_csvs`` are expanded recursively to
    all ``.csv`` files and deduplicated before scoring.
    """


    # archetyper splits text with nltk.sent_tokenize, which needs data that NLTK
    # doesn't ship. we grab it up front so that we don't fail mid-pipeline with
    # a wall of asterisks after the expensive steps have already run
    ensure_punkt(verbose=True)

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        out_features_csv = Path.cwd() / "features" / "archetypes" / analysis_ready.name
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)

    if not overwrite_existing and Path(out_features_csv).is_file():
        print("Archetypes output file already exists; returning existing file.")
        return out_features_csv


    # 2) resolve/validate the archetype CSVs
    # we allow passing either:
    #   • one or more CSV files, or
    #   • one or more directories containing CSVs (recursively).
    #
    # we lean on the shared find_files helper so we're not reinventing it here

    # 2) resolve/validate the archetype CSVs
    resolved_archetype_csvs: list[Path] = []

    for src in archetype_csvs:
        src_path = Path(src)
        if src_path.is_dir():
            # find all *.csv under this folder (recursive)
            found = find_files(
                root_dir=src_path,
                extensions=[".csv"],
                recursive=True,
                absolute=True,
                sort=True,
            )
            resolved_archetype_csvs.extend(Path(f) for f in found)
        else:
            resolved_archetype_csvs.append(src_path)

    # de-dup, normalize, and sort
    archetype_csvs = sorted({p.resolve() for p in resolved_archetype_csvs})

    if not archetype_csvs:
        raise ValueError(
            "No archetype CSVs found. Pass one or more CSV files, or a directory containing CSV files with your archetypes."
        )
    for p in archetype_csvs:
        if not p.exists():
            raise FileNotFoundError(f"Archetype CSV not found: {p}")



        # 3) stream (text_id, text, meta) → middle layer → features CSV
    def _iter_items_from_csv_with_meta(
        path: Path,
        *,
        id_col: str = "text_id",
        text_col: str = "text",
        wanted: Optional[Sequence[str]] = None,
    ) -> Iterable[Tuple[str, str, dict]]:
        """
        Stream (text_id, text, meta) from an analysis-ready CSV.

        Enforces that all requested `wanted` columns exist (fail fast).
        """
        wanted = list(wanted or [])
        with path.open("r", newline="", encoding=encoding) as f:
            reader = csv.DictReader(f, delimiter=delimiter)
            fields = reader.fieldnames or []
            if id_col not in fields or text_col not in fields:
                raise ValueError(
                    f"Expected columns '{id_col}' and '{text_col}' in {path}; found {fields}"
                )
            missing = [c for c in wanted if c not in fields]
            if missing:
                raise ValueError(
                    f"Requested id_cols not present in analysis-ready CSV {path}: {missing}"
                )

            for row in reader:
                tid = str(row.get(id_col, "") or "")
                txt = str(row.get(text_col, "") or "")
                meta = {c: str(row.get(c, "") or "") for c in wanted}
                # we name each tick because these rows are wildly uneven: one
                # book-sized document can take minutes where its neighbors
                # take milliseconds, and an unnamed pause that long looks
                # like a hang
                _ticker.tick(message=f"scoring {tid}")
                yield tid, txt, meta


    # the middle layer pulls the generator above lazily and writes as it goes,
    # so one yield is one row's worth of work handed over
    _ticker = Ticker(on_progress, count_rows(analysis_ready, on_progress=on_progress))

    # we import this late for the same reason the analyzer defers `archetypes`:
    # this module gets imported to read its signature far more often than it
    # actually runs
    from .dictionary_analyzers import multi_archetype_analyzer as maa

    # there's one shared rule for what rides along beside text_id -- see
    # resolve_passthrough_columns: same order of preference that every per-row
    # analyzer uses, and the same two columns that never get carried
    from ..helpers.row_map import resolve_passthrough_columns

    with analysis_ready.open("r", newline="", encoding=encoding) as _fh:
        _header = csv.DictReader(_fh).fieldnames or []
    passthrough = resolve_passthrough_columns(
        _header, id_cols=id_cols, group_by=group_by,
        analysis_ready=analysis_ready)

    maa.analyze_texts_to_csv(
        items=_iter_items_from_csv_with_meta(analysis_ready, wanted=passthrough),
        archetype_csvs=archetype_csvs,
        out_csv=out_features_csv,
        model_name=model_name,
        device=device,
        mean_center_vectors=mean_center_vectors,
        fisher_z_transform=fisher_z_transform,
        rounding=rounding,
        encoding=encoding,
        delimiter=delimiter,
        id_col_name="text_id",
        pass_through_cols=passthrough,  # these land right after text_id
        verbose=on_progress is None,
    )

    return out_features_csv

taters.text.analyze_with_dictionaries

analyze_with_dictionaries

analyze_with_dictionaries(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    on_progress=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    dict_paths,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    relative_freq=True,
    drop_punct=True,
    rounding=4,
    retain_captures=False,
    wildcard_mem=True
)

Compute LIWC-style dictionary features for text rows and write a wide features CSV.

The function supports exactly one of three input modes:

1. analysis_csv — Use a prebuilt file with columns text_id and text.
2. csv_path — Gather text from an arbitrary CSV using text_cols (and optional id_cols/group_by) to produce an analysis-ready file.
3. txt_dir — Gather text from a folder of .txt files.

If out_features_csv is omitted, the default output path is ./features/dictionary/<analysis_ready_filename>. Multiple dictionaries are supported; passing a directory discovers all .dic, .dicx, and .csv dictionary files recursively in a stable order. Global columns (e.g., word counts, punctuation) are emitted once (from the first dictionary) and each dictionary contributes a namespaced block.

Parameters:

Name	Type	Description	Default
csv_path	str or Path	Source CSV to gather from. Mutually exclusive with txt_dir and analysis_csv.	None
txt_dir	str or Path	Folder containing .txt files to gather from. Mutually exclusive with other modes.	None
analysis_csv	str or Path	Prebuilt analysis-ready CSV with exactly two columns: text_id and text.	None
gathered_csv	str or Path	Where to write the intermediate "analysis-ready" table built from csv_path or txt_dir. By default it lands beside the source -- which means analyzing a spreadsheet in someone's Downloads folder writes a file into their Downloads folder. Pass this to keep the intermediate with the rest of a run's output instead. Ignored when analysis_csv is given, because then no gathering happens.	None
on_progress	callable	Called as on_progress(done, total, message=None) so a UI can show a real bar instead of a spinner. Injected automatically by the pipeline runner for any step function that declares this parameter. See :mod:taters.helpers.progress for the contract.	None
out_features_csv	str or Path	Output file path. If None, defaults to ./features/dictionary/<analysis_ready_filename>.	None
overwrite_existing	bool	If False and the output file already exists, skip processing and return the path. This also controls the intermediate analysis-ready CSV: when True, it is rebuilt from the current source instead of reusing a stale copy from an earlier run.	False
dict_paths	Sequence[str or Path]	One or more dictionary inputs (files or directories). Supported extensions: .dic, .dicx, .csv. Directories are expanded recursively.	required
encoding	str	Text encoding used for reading/writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	When gathering from a CSV, name(s) of the column(s) containing text.	("text",)
id_cols	Sequence[str] or None	Optional ID columns to carry into grouping when gathering from CSV.	None
mode	(concat, separate)	Gathering behavior when multiple text columns are provided. "concat" joins them into one text field using joiner; "separate" creates one row per column.	"concat"
group_by	Sequence[str] or None	Optional grouping keys used during CSV gathering (e.g., ["speaker"]).	None
delimiter	str	Delimiter for reading/writing CSV files.	","
joiner	str	Separator used when concatenating multiple text chunks in "concat" mode.	" "
num_buckets	int	Number of temporary hash buckets used during scalable CSV gathering.	512
max_open_bucket_files	int	Maximum number of bucket files kept open concurrently during gathering.	64
tmp_root	str or Path or None	Root directory for temporary gathering artifacts.	None
recursive	bool	When gathering from a text folder, recurse into subdirectories.	True
pattern	str	Glob pattern for selecting text files when gathering from a folder.	"*.txt"
id_from	(stem, name, path)	How to derive text_id for gathered .txt files.	"stem"
include_source_path	bool	If True, include the absolute source path as an additional column when gathering from a text folder.	True
relative_freq	bool	Emit relative frequencies instead of raw counts, when supported by the dictionary engine.	True
drop_punct	bool	Drop punctuation prior to analysis (dictionary-dependent).	True
workers	int	Parallel processes for reading documents and scoring texts. 0 means automatic: three-quarters of the logical cores; 1 turns parallelism off. Output files are identical whatever the worker count.	0
rounding	int	Decimal places to round numeric outputs. Use None to disable rounding.	4
retain_captures	bool	Pass-through flag to the underlying analyzer to retain capture groups, if applicable.	False
wildcard_mem	bool	Pass-through optimization flag for wildcard handling in the analyzer.	True

Returns:

Type	Description
Path	Path to the written features CSV.

Raises:

Type	Description
FileNotFoundError	If input files/folders or any dictionary file cannot be found.
ValueError	If input modes are misconfigured (e.g., multiple sources provided or none), required columns are missing from the analysis-ready CSV, or unsupported dictionary extensions are encountered.

Examples:

Run on a transcript CSV, grouped by speaker:

>>> analyze_with_dictionaries(
...     csv_path="transcripts/session.csv",
...     text_cols=["text"], id_cols=["speaker"], group_by=["speaker"],
...     dict_paths=["dictionaries/liwc/LIWC-22 Dictionary (2022-01-27).dicx"]
... )
PosixPath('.../features/dictionary/session.csv')

Notes

If overwrite_existing is False and the output exists, the existing file path is returned without recomputation.

Source code in src\taters\text\analyze_with_dictionaries.py

@records_settings(binding=TEXT_INPUT, grain=TEXT_GRAIN,
                  outputs=("out_features_csv",),
                  # this is the word count that the categories are percentages of
                  bookkeeping=("WC",),
                  assets={"dict_paths": "dictionaries"})
def analyze_with_dictionaries(
    *,
    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[Union[str, Path]] = None,
    txt_dir: Optional[Union[str, Path]] = None,
    analysis_csv: Optional[Union[str, Path]] = None,  # if given, we skip gathering
    gathered_csv: Optional[Union[str, Path]] = None,
    on_progress: Optional[Callable[[int, int], None]] = None,

    # ----- Output -----
    out_features_csv: Optional[Union[str, Path]] = None,
    overwrite_existing: bool = False,
    workers: int = 0,  # if the file already exists, let's not overwrite by default

    # ----- Dictionaries -----
    dict_paths: Sequence[Union[str, Path]], # LIWC2007 (.dic) or LIWC-22 (.dicx, .csv)

    # ====== SHARED I/O OPTIONS ======
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS ======
    # these only matter when csv_path is provided
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[Union[str, Path]] = None,

    # ====== TXT FOLDER GATHER OPTIONS ======
    # these only matter when txt_dir is provided
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ====== ANALYZER OPTIONS (passed through to ContentCoder) ======
    relative_freq: bool = True,
    drop_punct: bool = True,
    rounding: int = 4,
    retain_captures: bool = False,
    wildcard_mem: bool = True,
) -> Path:
    """
    Compute LIWC-style dictionary features for text rows and write a wide features CSV.

    The function supports exactly one of three input modes:

    1. ``analysis_csv`` — Use a prebuilt file with columns ``text_id`` and ``text``.
    2. ``csv_path`` — Gather text from an arbitrary CSV using ``text_cols`` (and optional
    ``id_cols``/``group_by``) to produce an analysis-ready file.
    3. ``txt_dir`` — Gather text from a folder of ``.txt`` files.

    If ``out_features_csv`` is omitted, the default output path is
    ``./features/dictionary/<analysis_ready_filename>``. Multiple dictionaries are supported;
    passing a directory discovers all ``.dic``, ``.dicx``, and ``.csv`` dictionary files
    recursively in a stable order. Global columns (e.g., word counts, punctuation) are emitted
    once (from the first dictionary) and each dictionary contributes a namespaced block.

    Parameters
    ----------
    csv_path : str or pathlib.Path, optional
        Source CSV to gather from. Mutually exclusive with ``txt_dir`` and ``analysis_csv``.
    txt_dir : str or pathlib.Path, optional
        Folder containing ``.txt`` files to gather from. Mutually exclusive with other modes.
    analysis_csv : str or pathlib.Path, optional
        Prebuilt analysis-ready CSV with exactly two columns: ``text_id`` and ``text``.
    gathered_csv : str or pathlib.Path, optional
        Where to write the intermediate "analysis-ready" table built from
        ``csv_path`` or ``txt_dir``.

        By default it lands beside the *source* -- which means analyzing a
        spreadsheet in someone's Downloads folder writes a file into their
        Downloads folder. Pass this to keep the intermediate with the rest of a
        run's output instead. Ignored when ``analysis_csv`` is given, because
        then no gathering happens.
    on_progress : callable, optional
        Called as ``on_progress(done, total, message=None)`` so a UI can show a
        real bar instead of a spinner. Injected automatically by the pipeline
        runner for any step function that declares this parameter. See
        :mod:`taters.helpers.progress` for the contract.
    out_features_csv : str or pathlib.Path, optional
        Output file path. If ``None``, defaults to
        ``./features/dictionary/<analysis_ready_filename>``.
    overwrite_existing : bool, default=False
        If ``False`` and the output file already exists, skip processing and return the path.
        This also controls the intermediate analysis-ready CSV: when ``True``, it is rebuilt
        from the current source instead of reusing a stale copy from an earlier run.
    dict_paths : Sequence[str or pathlib.Path]
        One or more dictionary inputs (files or directories). Supported extensions:
        ``.dic``, ``.dicx``, ``.csv``. Directories are expanded recursively.
    encoding : str, default="utf-8-sig"
        Text encoding used for reading/writing CSV files.
    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV, name(s) of the column(s) containing text.
    id_cols : Sequence[str] or None, optional
        Optional ID columns to carry into grouping when gathering from CSV.
    mode : {"concat", "separate"}, default="concat"
        Gathering behavior when multiple text columns are provided. ``"concat"`` joins them
        into one text field using ``joiner``; ``"separate"`` creates one row per column.
    group_by : Sequence[str] or None, optional
        Optional grouping keys used during CSV gathering (e.g., ``["speaker"]``).
    delimiter : str, default=","
        Delimiter for reading/writing CSV files.
    joiner : str, default=" "
        Separator used when concatenating multiple text chunks in ``"concat"`` mode.
    num_buckets : int, default=512
        Number of temporary hash buckets used during scalable CSV gathering.
    max_open_bucket_files : int, default=64
        Maximum number of bucket files kept open concurrently during gathering.
    tmp_root : str or pathlib.Path or None, optional
        Root directory for temporary gathering artifacts.
    recursive : bool, default=True
        When gathering from a text folder, recurse into subdirectories.
    pattern : str, default="*.txt"
        Glob pattern for selecting text files when gathering from a folder.
    id_from : {"stem", "name", "path"}, default="stem"
        How to derive ``text_id`` for gathered ``.txt`` files.
    include_source_path : bool, default=True
        If ``True``, include the absolute source path as an additional column when gathering
        from a text folder.
    relative_freq : bool, default=True
        Emit relative frequencies instead of raw counts, when supported by the dictionary engine.
    drop_punct : bool, default=True
        Drop punctuation prior to analysis (dictionary-dependent).
    workers : int, default=0
        Parallel processes for reading documents and scoring texts. ``0`` means automatic:
        three-quarters of the logical cores; ``1`` turns parallelism off. Output files are
        identical whatever the worker count.
    rounding : int, default=4
        Decimal places to round numeric outputs. Use ``None`` to disable rounding.
    retain_captures : bool, default=False
        Pass-through flag to the underlying analyzer to retain capture groups, if applicable.
    wildcard_mem : bool, default=True
        Pass-through optimization flag for wildcard handling in the analyzer.

    Returns
    -------
    pathlib.Path
        Path to the written features CSV.

    Raises
    ------
    FileNotFoundError
        If input files/folders or any dictionary file cannot be found.
    ValueError
        If input modes are misconfigured (e.g., multiple sources provided or none),
        required columns are missing from the analysis-ready CSV, or unsupported
        dictionary extensions are encountered.

    Examples
    --------
    Run on a transcript CSV, grouped by speaker:

    >>> analyze_with_dictionaries(
    ...     csv_path="transcripts/session.csv",
    ...     text_cols=["text"], id_cols=["speaker"], group_by=["speaker"],
    ...     dict_paths=["dictionaries/liwc/LIWC-22 Dictionary (2022-01-27).dicx"]
    ... )
    PosixPath('.../features/dictionary/session.csv')

    Notes
    -----
    If ``overwrite_existing`` is ``False`` and the output exists, the existing file path
    is returned without recomputation.
    """


    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        out_features_csv = Path.cwd() / "features" / "dictionary" / analysis_ready.name
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)

    if not overwrite_existing and Path(out_features_csv).is_file():
        print("Dictionary content coding output file already exists; returning existing file.")
        return out_features_csv


    # 2) validate the dictionaries
    def _expand_dict_inputs(paths):
        """
        Normalize dictionary inputs into a unique, ordered list of files.

        Parameters
        ----------
        paths : Iterable[Union[str, pathlib.Path]]
            Files or directories. Directories are expanded recursively to files with
            extensions ``.dic``, ``.dicx``, or ``.csv``.

        Returns
        -------
        list[pathlib.Path]
            Deduplicated, resolved file paths in stable order.

        Raises
        ------
        FileNotFoundError
            If a referenced file or directory does not exist.
        ValueError
            If a file has an unsupported extension or if no dictionary files are found.
        """

        out = []
        seen = set()
        for p in map(Path, paths):
            if p.is_dir():
                # find .dic/.dicx/.csv under this folder (recursive), stable order
                found = find_files(
                    root_dir=p,
                    extensions=[".dic", ".dicx", ".csv"],
                    recursive=True,
                    absolute=True,
                    sort=True,
                )
                for f in found:
                    fp = Path(f).resolve()
                    if fp.suffix.lower().lstrip(".") in {"dic", "dicx", "csv"}:
                        if fp not in seen:
                            out.append(fp)
                            seen.add(fp)
            else:
                if not p.exists():
                    raise FileNotFoundError(f"Dictionary path not found: {p}")
                fp = p.resolve()
                if fp.suffix.lower().lstrip(".") not in {"dic", "dicx", "csv"}:
                    raise ValueError(f"Unsupported dictionary extension: {fp.name}")
                if fp not in seen:
                    out.append(fp)
                    seen.add(fp)
        if not out:
            raise ValueError("No dictionary files found. Supply .dic/.dicx/.csv files or folders containing them.")
        return out

    dict_paths = _expand_dict_inputs(dict_paths)

        # 3) stream the analysis-ready CSV into the analyzer → features CSV
    def _iter_items_from_csv_with_meta(
        path: Path,
        *,
        id_col: str = "text_id",
        text_col: str = "text",
        pass_through_cols: Optional[Sequence[str]] = None,
    ) -> Iterable[Tuple[str, str, dict]]:
        """
        Stream (text_id, text, meta) from an analysis-ready CSV.

        Parameters
        ----------
        path : pathlib.Path
            Path to the analysis-ready CSV file.
        id_col : str, default="text_id"
            Identifier column.
        text_col : str, default="text"
            Text column.
        pass_through_cols : Sequence[str] or None
            Extra columns to fetch per row and forward to the analyzer.

        Yields
        ------
        tuple[str, str, dict]
            (text_id, text, meta_dict) where meta_dict maps each pass-through column
            to its string value ('' if missing).
        """
        wanted = list(pass_through_cols or [])
        with path.open("r", newline="", encoding=encoding) as f:
            reader = csv.DictReader(f, delimiter=delimiter)
            fields = reader.fieldnames or []
            if id_col not in fields or text_col not in fields:
                raise ValueError(
                    f"Expected columns '{id_col}' and '{text_col}' in {path}; found {fields}"
                )
            # if id_cols were requested, make sure they exist up-front (fail fast)
            missing = [c for c in wanted if c not in fields]
            if missing:
                raise ValueError(
                    f"Requested id_cols not present in analysis-ready CSV {path}: {missing}"
                )

            for row in reader:
                tid = str(row.get(id_col, "") or "")
                text = str(row.get(text_col, "") or "")
                meta = {c: str(row.get(c, "") or "") for c in wanted}
                yield tid, text, meta


    # we use multi_dict_analyzer as the middle layer (new API). it pulls the
    # generator above lazily, writes as it goes, and owns the progress story:
    # "scoring documents", with one sub-bar per document in flight on displays
    # that can show it
    total_rows = count_rows(analysis_ready, on_progress=on_progress)

    # there's one shared rule for what rides along beside text_id -- see
    # resolve_passthrough_columns: same order of preference that every per-row
    # analyzer uses, and the same two columns that never get carried
    from ..helpers.row_map import resolve_passthrough_columns

    with analysis_ready.open("r", newline="", encoding=encoding) as _fh:
        _header = csv.DictReader(_fh).fieldnames or []
    passthrough = resolve_passthrough_columns(
        _header, id_cols=id_cols, group_by=group_by,
        analysis_ready=analysis_ready)

    mda.analyze_texts_to_csv(
        items=_iter_items_from_csv_with_meta(analysis_ready, pass_through_cols=passthrough),
        dict_files=dict_paths,
        out_csv=out_features_csv,
        relative_freq=relative_freq,
        drop_punct=drop_punct,
        rounding=rounding,
        retain_captures=retain_captures,
        wildcard_mem=wildcard_mem,
        id_col_name="text_id",
        pass_through_cols=passthrough,  # these land right after text_id
        workers=workers,
        on_progress=on_progress,
        total_hint=total_rows,
        encoding=encoding,
        verbose=on_progress is None,
    )


    return out_features_csv

taters.text.analyze_entropy

Entropy and diversity, one row per document.

Lexical richness (:mod:analyze_lexical_richness) already answers "how varied is the vocabulary" with a dozen indices. This answers the same family of questions from information theory instead, which buys three things those indices do not give you.

One family rather than a dozen names. Type-token ratio, Simpson's D and Yule's K are not unrelated measures; they are points on one curve, the Rényi entropies, indexed by an order q that says how much weight to put on common words versus rare ones. At q = 0 the answer is the number of types, at 1 it is Shannon, at 2 it is Simpson, and as q grows it approaches the commonest word's share alone. So the order is reported as a setting of the measure rather than buried in a formula named after somebody, and you can read the profile across orders instead of picking one index and hoping.

An honest answer on short texts. Plug-in entropy is biased downward, and the bias depends on how many tokens you had -- worse than type-token ratio, which is the usual cautionary example. A 50-word answer and a 5,000-word essay are not comparable on the plug-in number even when their vocabularies are equally varied. Four bias corrections are computed beside it (Miller-Madow, Chao-Shen, Grassberger, NSB), so the difference between them is visible rather than assumed away. Where they disagree, the text was too short to say.

Structure as well as variety. Entropy over single tokens measures how varied the vocabulary is. Conditional entropy over pairs and triples measures how predictable the next token is given the last one or two, which is a different construct -- a text can have a wide vocabulary and be highly formulaic. The compression ratios are a crude estimate of the same quantity that makes no assumption about tokenization at all.

Everything is computed over two units: words, and characters. Character-level measures need no tokenizer and survive languages the word tokenizer handles badly.

References

• Hill, M. O. (1973). Diversity and evenness: a unifying notation and its consequences. Ecology, 54(2), 427-432.
• Rényi, A. (1961). On measures of entropy and information. Berkeley Symposium on Mathematical Statistics and Probability.
• Tsallis, C. (1988). Possible generalization of Boltzmann-Gibbs statistics. Journal of Statistical Physics, 52, 479-487.
• Miller, G. A. (1955). Note on the bias of information estimates. Information Theory in Psychology.
• Chao, A., & Shen, T.-J. (2003). Nonparametric estimation of Shannon's index of diversity when there are unseen species. Environmental and Ecological Statistics, 10, 429-443.
• Grassberger, P. (2003). Entropy estimates from insufficient samplings. arXiv:physics/0307138.
• Nemenman, I., Shafee, F., & Bialek, W. (2002). Entropy and inference, revisited. NIPS 14.
• Pielou, E. C. (1966). The measurement of diversity in different types of biological collections. Journal of Theoretical Biology, 13, 131-144.

analyze_entropy

analyze_entropy(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    out_features_csv=None,
    overwrite_existing=False,
    lowercase=True,
    strip_punctuation=True,
    strip_digits=True,
    max_order=DEFAULT_MAX_ORDER,
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    encoding="utf-8-sig",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=False,
    pass_through_cols=None,
    workers=0,
    on_progress=None,
    verbose=True
)

Entropy and diversity measures for each text, over words and characters.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The usual input contract: a spreadsheet of texts, a folder of documents, or a table somebody already gathered.	None
txt_dir	Optional[PathLike]	The usual input contract: a spreadsheet of texts, a folder of documents, or a table somebody already gathered.	None
analysis_csv	Optional[PathLike]	The usual input contract: a spreadsheet of texts, a folder of documents, or a table somebody already gathered.	None
gathered_csv	Optional[PathLike]	The usual input contract: a spreadsheet of texts, a folder of documents, or a table somebody already gathered.	None
out_features_csv	str or Path	Where to write. Default ./features/entropy.csv.	None
overwrite_existing	bool	Rebuild the table if it is already there.	False
lowercase	bool	Fold case before counting. Off, "The" and "the" are two types.	True
strip_punctuation	bool	Applied to the word units only, matching what analyze_lexical_richness does, so the two are comparable. The character units always keep punctuation, since that is most of what distinguishes character-level style.	True
strip_digits	bool	Applied to the word units only, matching what analyze_lexical_richness does, so the two are comparable. The character units always keep punctuation, since that is most of what distinguishes character-level style.	True
max_order	int	How far the block and conditional entropies go. Order 3 over characters already wants a few thousand characters to be worth reading; higher orders on short texts measure the sample rather than the text.	3
text_cols	sequence of str	Which spreadsheet columns hold the text, when one has to be gathered.	``("text",)``
id_cols	sequence of str	Columns that compose each row's identifier.	None
mode	('concat', 'separate')	Measure several text columns joined together, or one at a time.	"concat"
group_by	sequence of str	Combine rows sharing these columns before measuring.	None
delimiter	str	The gatherer's own settings: how to read the spreadsheet, what to join combined texts with, and how much to spill to disk on a large one.	','
encoding	str	The gatherer's own settings: how to read the spreadsheet, what to join combined texts with, and how much to spill to disk on a large one.	','
joiner	str	The gatherer's own settings: how to read the spreadsheet, what to join combined texts with, and how much to spill to disk on a large one.	','
num_buckets	str	The gatherer's own settings: how to read the spreadsheet, what to join combined texts with, and how much to spill to disk on a large one.	','
max_open_bucket_files	str	The gatherer's own settings: how to read the spreadsheet, what to join combined texts with, and how much to spill to disk on a large one.	','
tmp_root	str	The gatherer's own settings: how to read the spreadsheet, what to join combined texts with, and how much to spill to disk on a large one.	','
recursive	bool	Search subfolders when the input is a folder of documents.	True
pattern	str	Which files in that folder count as documents.	DOCUMENT_PATTERN
id_from	('stem', 'name', 'path')	What to call each document, when the input is a folder.	"stem"
include_source_path	bool	Carry each document's path into the gathered table.	False
pass_through_cols	sequence of str	Columns of the gathered table to copy into the output.	None
workers	int	Processes. 0 picks a sensible number for the job's size.	0

Returns:

Type	Description
Path	out_features_csv.

Notes

Read coverage before anything else. It says what share of the distribution the text actually showed you, and when it is low the four Shannon estimates will disagree -- that disagreement is the honest width of the answer, not noise to average away.

Every entropy is reported in bits and again as an effective number of types, which is the same number in units people can hold in their head.

A conditional entropy can come out slightly negative on a short text. That is impossible in truth and is the estimate telling on itself: the block entropy above it is more undersampled than the one below, so the difference goes the wrong way. It is left as it falls rather than clamped at zero, because a negative number is a visible sign that the text was too short for that order and a zero is not.

Source code in src\taters\text\analyze_entropy.py

@records_settings(binding=TEXT_INPUT, grain=TEXT_GRAIN,
                  outputs=("out_features_csv",), bookkeeping=BOOKKEEPING)
def analyze_entropy(
    *,
    # ----- input: the same three ways every text step takes one -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,

    # ----- output -----
    out_features_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,

    # ----- how to cut the text up -----
    lowercase: bool = True,
    strip_punctuation: bool = True,
    strip_digits: bool = True,
    max_order: int = DEFAULT_MAX_ORDER,

    # ----- gathering, for when the input is a spreadsheet or a folder -----
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    encoding: str = "utf-8-sig",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = False,
    pass_through_cols: Optional[Sequence[str]] = None,

    workers: int = 0,
    on_progress: Optional[Callable[..., None]] = None,
    verbose: bool = True,
) -> Path:
    """
    Entropy and diversity measures for each text, over words and characters.

    Parameters
    ----------
    csv_path, txt_dir, analysis_csv, gathered_csv
        The usual input contract: a spreadsheet of texts, a folder of
        documents, or a table somebody already gathered.
    out_features_csv : str or Path, optional
        Where to write. Default ``./features/entropy.csv``.
    overwrite_existing : bool, default False
        Rebuild the table if it is already there.
    lowercase : bool, default True
        Fold case before counting. Off, "The" and "the" are two types.
    strip_punctuation, strip_digits : bool, default True
        Applied to the *word* units only, matching what
        `analyze_lexical_richness` does, so the two are comparable. The
        character units always keep punctuation, since that is most of what
        distinguishes character-level style.
    max_order : int, default 3
        How far the block and conditional entropies go. Order 3 over
        characters already wants a few thousand characters to be worth
        reading; higher orders on short texts measure the sample rather than
        the text.
    text_cols : sequence of str, default ``("text",)``
        Which spreadsheet columns hold the text, when one has to be gathered.
    id_cols : sequence of str, optional
        Columns that compose each row's identifier.
    mode : {"concat", "separate"}, default "concat"
        Measure several text columns joined together, or one at a time.
    group_by : sequence of str, optional
        Combine rows sharing these columns before measuring.
    delimiter, encoding, joiner, num_buckets, max_open_bucket_files, tmp_root
        The gatherer's own settings: how to read the spreadsheet, what to join
        combined texts with, and how much to spill to disk on a large one.
    recursive : bool, default True
        Search subfolders when the input is a folder of documents.
    pattern : str
        Which files in that folder count as documents.
    id_from : {"stem", "name", "path"}, default "stem"
        What to call each document, when the input is a folder.
    include_source_path : bool, default False
        Carry each document's path into the gathered table.
    pass_through_cols : sequence of str, optional
        Columns of the gathered table to copy into the output.
    workers : int, default 0
        Processes. 0 picks a sensible number for the job's size.

    Returns
    -------
    pathlib.Path
        ``out_features_csv``.

    Notes
    -----
    Read `coverage` before anything else. It says what share of the
    distribution the text actually showed you, and when it is low the four
    Shannon estimates will disagree -- that disagreement is the honest width
    of the answer, not noise to average away.

    Every entropy is reported in bits and again as an effective number of
    types, which is the same number in units people can hold in their head.

    A conditional entropy can come out slightly *negative* on a short text.
    That is impossible in truth and is the estimate telling on itself: the
    block entropy above it is more undersampled than the one below, so the
    difference goes the wrong way. It is left as it falls rather than clamped
    at zero, because a negative number is a visible sign that the text was too
    short for that order and a zero is not.
    """
    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers, verbose=verbose)

    out = Path(out_features_csv) if out_features_csv \
        else Path("features") / "entropy.csv"
    if out.exists() and not overwrite_existing:
        if verbose:
            print(f"[entropy] exists, leaving alone: {out}")
        return out
    out.parent.mkdir(parents=True, exist_ok=True)

    if int(max_order) < 1:
        raise ValueError("max_order must be at least 1: order 1 is the "
                         "single-unit entropy, which is the point of it.")

    with Path(analysis_ready).open("r", newline="", encoding=encoding) as fin:
        header_fields = csv.DictReader(fin).fieldnames or []
    if "text_id" not in header_fields or "text" not in header_fields:
        raise ValueError(f"Expected columns 'text_id' and 'text' in "
                         f"{analysis_ready}; found {header_fields}")
    # one shared rule for what rides along beside text_id
    passthrough = resolve_passthrough_columns(
        header_fields, pass_through_cols=pass_through_cols, id_cols=id_cols,
        group_by=group_by, analysis_ready=analysis_ready)
    names = _measure_names(int(max_order))
    scorer = _Scorer(lowercase=lowercase, strip_punctuation=strip_punctuation,
                     strip_digits=strip_digits, max_order=int(max_order))

    announce(on_progress, "measuring entropy")
    with atomic_write(out, newline="", encoding="utf-8") as fout:
        writer = csv.DictWriter(fout, fieldnames=["text_id", *passthrough,
                                                  *names])
        writer.writeheader()
        for row, values in map_text_rows(
                analysis_ready, encoding=encoding, workers=workers,
                message="entropy", on_progress=on_progress,
                inline_fn=scorer, pool_fn=scorer):
            out_row = {"text_id": row.get("text_id", "")}
            for col in passthrough:
                out_row[col] = row.get(col, "")
            for name in names:
                value = values.get(name, "")
                out_row[name] = (f"{value:.6g}" if isinstance(value, float)
                                 else value)
            writer.writerow(out_row)

    if verbose:
        print(f"[entropy] wrote {out}")
    return out

berger_parker

berger_parker(counts)

The commonest unit's share -- dominance, the q -> infinity end.

Source code in src\taters\text\analyze_entropy.py

def berger_parker(counts: Sequence[int]) -> float:
    """The commonest unit's share -- dominance, the q -> infinity end."""
    total = float(sum(counts))
    return float(max(counts) / total) if total > 0 else 0.0

chao1

chao1(counts)

How many types the text would have had if you had kept reading.

The bias-corrected Chao1 estimator: observed types plus a term built from how many appeared once and twice. Reported in its own right -- it is the q = 0 diversity you cannot see -- and used as the alphabet size NSB needs.

Source code in src\taters\text\analyze_entropy.py

def chao1(counts: Sequence[int]) -> float:
    """
    How many types the text would have had if you had kept reading.

    The bias-corrected Chao1 estimator: observed types plus a term built from
    how many appeared once and twice. Reported in its own right -- it is the
    q = 0 diversity you cannot see -- and used as the alphabet size NSB needs.
    """
    seen = float(len(counts))
    f1 = float(sum(1 for c in counts if c == 1))
    f2 = float(sum(1 for c in counts if c == 2))
    return seen + (f1 * (f1 - 1.0)) / (2.0 * (f2 + 1.0))

chao_shen_bits

chao_shen_bits(counts)

Coverage-adjusted, after Chao and Shen (2003).

Estimates what share of the distribution you actually saw -- from how many words appeared exactly once -- and reweights accordingly. The one that holds up best on the short, Zipfian samples that text usually is.

Source code in src\taters\text\analyze_entropy.py

def chao_shen_bits(counts: Sequence[int]) -> float:
    """
    Coverage-adjusted, after Chao and Shen (2003).

    Estimates what share of the distribution you actually saw -- from how many
    words appeared exactly once -- and reweights accordingly. The one that
    holds up best on the short, Zipfian samples that text usually is.
    """
    total = float(sum(counts))
    if total <= 0:
        return 0.0
    singles = float(sum(1 for c in counts if c == 1))
    if singles >= total:
        # every word appeared once, so coverage estimates as zero and the
        # formula divides by it. back off by one rather than refuse.
        singles = total - 1.0
    coverage = 1.0 - singles / total
    if coverage <= 0.0:
        return shannon_bits(counts)
    out = 0.0
    for c in counts:
        p = coverage * c / total
        seen = 1.0 - (1.0 - p) ** total
        if seen > 0.0:
            out -= p * log(p) / seen
    return max(0.0, float(out / LN2))

coverage

coverage(counts)

Good-Turing sample coverage: the share of the distribution the text actually showed you. Low coverage is the signal that the entropy numbers below it are estimates rather than measurements.

Source code in src\taters\text\analyze_entropy.py

def coverage(counts: Sequence[int]) -> float:
    """Good-Turing sample coverage: the share of the distribution the text
    actually showed you. Low coverage is the signal that the entropy numbers
    below it are estimates rather than measurements."""
    total = float(sum(counts))
    if total <= 0:
        return 0.0
    singles = float(sum(1 for c in counts if c == 1))
    return float(max(0.0, 1.0 - singles / total))

grassberger_bits

grassberger_bits(counts)

Grassberger (2003), which corrects each count by a digamma term.

Source code in src\taters\text\analyze_entropy.py

def grassberger_bits(counts: Sequence[int]) -> float:
    """Grassberger (2003), which corrects each count by a digamma term."""
    from scipy.special import digamma

    total = float(sum(counts))
    if total <= 0:
        return 0.0
    out = 0.0
    for c in counts:
        g = digamma(c) + 0.5 * ((-1.0) ** c) * (digamma((c + 1) / 2.0)
                                                - digamma(c / 2.0))
        out += c * g
    return max(0.0, float((log(total) - out / total) / LN2))

hill_number

hill_number(counts, order)

The Rényi entropy as an effective number of types.

Reported alongside every entropy because "4.2 bits" is not a quantity anybody has intuitions about and "18 equally common words" is.

Source code in src\taters\text\analyze_entropy.py

def hill_number(counts: Sequence[int], order: float) -> float:
    """
    The Rényi entropy as an effective number of types.

    Reported alongside every entropy because "4.2 bits" is not a quantity
    anybody has intuitions about and "18 equally common words" is.
    """
    return float(2.0 ** renyi_bits(counts, order))

miller_madow_bits

miller_madow_bits(counts)

Plug-in plus (V - 1) / 2N: the leading term of the bias.

The cheapest correction and the least effective on a badly undersampled text, because it only knows how many types you saw.

Source code in src\taters\text\analyze_entropy.py

def miller_madow_bits(counts: Sequence[int]) -> float:
    """
    Plug-in plus ``(V - 1) / 2N``: the leading term of the bias.

    The cheapest correction and the least effective on a badly undersampled
    text, because it only knows how many types you *saw*.
    """
    total = float(sum(counts))
    if total <= 0:
        return 0.0
    return max(0.0, shannon_bits(counts)
               + (len(counts) - 1) / (2.0 * total * LN2))

nsb_bits

nsb_bits(counts, alphabet=None)

Nemenman-Shafee-Bialek: Bayesian, integrated over the Dirichlet prior.

The only estimator here that needs to be told how many types the text could have used, because its prior is over distributions on a known alphabet. Text has no such number, and the answer moves by three bits across plausible guesses -- more than any two other estimators differ -- so guessing badly is worse than not using it.

So the guess is not left to anybody: alphabet defaults to :func:chao1, which estimates the unseen types from the seen ones. That makes it self-tuning and, on samples where the truth is known, accurate to within a tenth of a bit.

Source code in src\taters\text\analyze_entropy.py

def nsb_bits(counts: Sequence[int], alphabet: Optional[int] = None) -> float:
    """
    Nemenman-Shafee-Bialek: Bayesian, integrated over the Dirichlet prior.

    The only estimator here that needs to be told how many types the text
    *could* have used, because its prior is over distributions on a known
    alphabet. Text has no such number, and the answer moves by three bits
    across plausible guesses -- more than any two other estimators differ --
    so guessing badly is worse than not using it.

    So the guess is not left to anybody: ``alphabet`` defaults to
    :func:`chao1`, which estimates the unseen types from the seen ones. That
    makes it self-tuning and, on samples where the truth is known, accurate to
    within a tenth of a bit.
    """
    import numpy as np
    from scipy.special import digamma, gammaln, polygamma

    n = np.asarray(list(counts), dtype=float)
    total = float(n.sum())
    if total <= 0 or len(n) <= 1:
        return 0.0
    size = int(max(len(n), round(alphabet if alphabet else chao1(counts))))
    if size <= 1:
        return 0.0

    betas = np.exp(np.linspace(log(1e-4), log(1e3), 400))
    weights = np.empty_like(betas)
    means = np.empty_like(betas)
    for i, b in enumerate(betas):
        # the likelihood of this concentration, and NSB's prior on it (chosen
        # so the implied prior on the entropy itself is close to flat)
        like = (gammaln(size * b) - gammaln(total + size * b)
                + float((gammaln(n + b) - gammaln(b)).sum()))
        slope = size * polygamma(1, size * b + 1) - polygamma(1, b + 1)
        weights[i] = like + log(max(slope, 1e-300))
        seen = float(((n + b) / (total + size * b)
                      * digamma(n + b + 1)).sum())
        # the types that never turned up still carry prior weight. leaving
        # them out let the estimate climb above log2(alphabet), which is
        # impossible.
        unseen = (size - len(n)) * (b / (total + size * b)) * digamma(b + 1)
        means[i] = digamma(total + size * b + 1) - seen - unseen
    weights -= weights.max()
    # times beta, because the grid is spaced in log(beta)
    w = np.exp(weights) * betas
    if not w.sum():
        return shannon_bits(counts)
    return float((w * means).sum() / w.sum() / LN2)

renyi_bits

renyi_bits(counts, order)

Rényi entropy of the given order, in bits. Order 1 is Shannon.

Source code in src\taters\text\analyze_entropy.py

def renyi_bits(counts: Sequence[int], order: float) -> float:
    """Rényi entropy of the given order, in bits. Order 1 is Shannon."""
    total = float(sum(counts))
    if total <= 0:
        return 0.0
    ps = [c / total for c in counts]
    if order == 1.0:
        return shannon_bits(counts)
    if order == 0.0:
        return log2(len(ps))
    if order == float("inf"):
        return -log2(max(ps))
    return float(log(sum(p ** order for p in ps)) / (1.0 - order) / LN2)

shannon_bits

shannon_bits(counts)

Plug-in (maximum likelihood) Shannon entropy, in bits.

Source code in src\taters\text\analyze_entropy.py

def shannon_bits(counts: Sequence[int]) -> float:
    """Plug-in (maximum likelihood) Shannon entropy, in bits."""
    total = float(sum(counts))
    if total <= 0:
        return 0.0
    # clamped at zero: one repeated word gives -0.0, and a rounding wobble can
    # give a tiny negative. an entropy is never below zero.
    return max(0.0, float(-sum((c / total) * log(c / total)
                               for c in counts) / LN2))

tsallis

tsallis(counts, order)

Tsallis entropy: the same family under a different, non-logarithmic way of combining independent parts.

Source code in src\taters\text\analyze_entropy.py

def tsallis(counts: Sequence[int], order: float) -> float:
    """Tsallis entropy: the same family under a different, non-logarithmic
    way of combining independent parts."""
    total = float(sum(counts))
    if total <= 0 or order == 1.0:
        return shannon_bits(counts) * LN2
    ps = [c / total for c in counts]
    return float((1.0 - sum(p ** order for p in ps)) / (order - 1.0))

taters.text.analyze_lexical_richness

analyze_lexical_richness

analyze_lexical_richness(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    on_progress=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    msttr_window=100,
    mattr_window=100,
    mtld_threshold=0.72,
    hdd_draws=42,
    vocd_ntokens=50,
    vocd_within_sample=100,
    vocd_iterations=3,
    vocd_seed=42,
    pass_through_cols=None
)

Compute lexical richness/diversity metrics for each text row and write a features CSV. Draws heavily from https://github.com/LSYS/lexicalrichness but makes several key changes with the goals of minimizing dependencies, attempting to make some speed optimizations with grid search instead of precise curve specifications, and making some principled decisions around punctuation/hyphenization that differ from the original Note that these decisions are not objectively "better" than the original but, instead, reflect my own experiences/intuitions about what makes sense.

This function accepts (a) an analysis-ready CSV (with columns text_id,text), (b) a raw CSV plus instructions for gathering/aggregation, or (c) a folder of .txt files. For each resulting row of text, it tokenizes words and computes a suite of classical lexical richness measures (e.g., TTR, Herdan's C, Yule's K, MTLD, MATTR, HDD, VOCD). Results are written as a wide CSV whose rows align with the rows in the analysis-ready table (or the gathered group_by rows), preserving any non-text metadata columns.

Parameters:

Name	Type	Description	Default
csv_path	str or Path	Source CSV to gather from. Use with text_cols, optional id_cols, and optional group_by. Exactly one of csv_path, txt_dir, or analysis_csv must be provided (unless analysis_csv is given, which skips gathering).	None
txt_dir	str or Path	Folder of .txt files to gather. File identifiers are created from filenames via id_from and (optionally) a source_path column when include_source_path=True.	None
analysis_csv	str or Path	Existing analysis-ready CSV with columns text_id,text. When provided, all gathering options are ignored and the file is used as-is.	None
gathered_csv	str or Path	Where to write the intermediate "analysis-ready" table built from csv_path or txt_dir. By default it lands beside the source -- which means analyzing a spreadsheet in someone's Downloads folder writes a file into their Downloads folder. Pass this to keep the intermediate with the rest of a run's output instead. Ignored when analysis_csv is given, because then no gathering happens.	None
on_progress	callable	Called as on_progress(done, total, message=None) so a UI can show a real bar instead of a spinner. total is the row count of the analysis-ready table, which is fully written before measuring starts, so it is known up front. Until it is, total is None and done is a running tally -- the long silent passes (reading the input, counting its rows) report through the same callback with a message saying which one is running. Injected automatically by the pipeline runner for any step function that declares this parameter.	None
out_features_csv	str or Path	Output CSV path. If omitted, defaults to ./features/lexical-richness/<analysis_ready_filename>.	None
overwrite_existing	bool	If False and out_features_csv exists, the function short-circuits and returns the existing path without recomputation. This also controls the intermediate analysis-ready CSV: when True, it is rebuilt from the current source instead of reusing a stale copy from an earlier run.	False
encoding	str	Encoding for reading/writing CSVs.	"utf-8-sig"
text_cols	sequence of str	Text column(s) to use when csv_path is provided. When multiple columns are given, they are combined according to mode (concat or separate).	("text",)
id_cols	sequence of str	Columns to carry through unchanged into the analysis-ready CSV prior to analysis (e.g., ["source","speaker"]). These will also appear in the output features CSV.	None
mode	('concat', 'separate')	Gathering behavior when multiple text_cols are provided. "concat" joins values using joiner; "separate" produces separate rows per text column.	"concat"
group_by	sequence of str	If provided, texts are grouped by these columns before analysis (e.g., ["source","speaker"]). With mode="concat", all texts in a group are joined into one blob per group; with mode="separate", they remain separate rows.	None
delimiter	str	Column separator of the input spreadsheet. The gathered table and the output are always comma-separated.	","
joiner	str	String used to join text fields when mode="concat".	" "
num_buckets	int	Internal streaming/gather parameter to control temporary file bucketing (passed through to the gatherer).	512
max_open_bucket_files	int	Maximum number of temporary files simultaneously open during gathering.	64
tmp_root	str or Path	Temporary directory root for the gatherer. Defaults to a system temp location.	None
recursive	bool	When txt_dir is provided, whether to search subdirectories for .txt files.	True
pattern	str	Glob pattern for discovering text files under txt_dir.	"*.txt"
id_from	('stem', 'name', 'path')	How to construct text_id for .txt inputs: file stem, full name, or relative path.	"stem"
include_source_path	bool	When txt_dir is used, include a source_path column in the analysis-ready CSV.	True
msttr_window	int	Window size for MSTTR (Mean Segmental TTR). Must be smaller than the number of tokens in the text to produce a value.	100
mattr_window	int	Window size for MATTR (Moving-Average TTR). Must be smaller than the number of tokens.	100
mtld_threshold	float	MTLD threshold for factor completion. A higher threshold yields shorter factors and typically lower MTLD values; the default follows common practice.	0.72
hdd_draws	int	Sample size n for HD-D (Hypergeometric Distribution Diversity). Must be less than the number of tokens to produce a value.	42
vocd_ntokens	int	Maximum sample size used to estimate VOCD (D). For each N in 35..vocd_ntokens, the function computes the average TTR over many random samples (vocd_within_sample).	50
vocd_within_sample	int	Number of random samples drawn per N when estimating VOCD.	100
vocd_iterations	int	Repeat-estimate count for VOCD. The best-fit D from each repetition is averaged.	3
vocd_seed	int	Seed for the VOCD random sampler (controls reproducibility across runs).	42
pass_through_cols	Sequence[str] or None	Extra input columns to copy into the output beside text_id.	None
workers	int	Parallel processes for reading documents. 0 means automatic: three-quarters of the logical cores; 1 turns parallelism off. Output files are identical whatever the worker count.	0

Returns:

Type	Description
Path	Path to the written features CSV.

Output shape

The features CSV starts with::

text_id, <pass-through columns...>, ttr, rttr, cttr, ...

Pass-through behavior:

• If pass_through_cols is provided, those columns are included in that order.
• Otherwise, if id_cols were used during gathering, they are included in that order.
• Otherwise (backward compatible), all non-text columns from the analysis-ready CSV are passed through.

Metrics emitted per row (None if the text is too short): ttr, rttr, cttr, herdan_c, summer_s, dugast, maas, yule_k, yule_i, herdan_vm, simpson_d, msttr_{msttr_window}, mattr_{mattr_window}, mtld_{mtld_threshold}, hdd_{hdd_draws}, vocd_{vocd_ntokens}.

Notes

Tokenization and preprocessing. Texts are lowercased, digits are removed, and punctuation characters are replaced with spaces prior to tokenization. As a result, hyphenated forms such as "state-of-the-art" will be split into separate tokens ("state", "of", "the", "art"). This choice yields robust behavior across corpora but can produce different numeric results than implementations that remove hyphens (treating "state-of-the-art" as a single token). If you require strict parity with a hyphen-removal scheme, adapt the internal preprocessing accordingly.

Metrics. The following measures are emitted per row (values are None when a text is too short to support the computation): - ttr: Type-Token Ratio (|V| / N) - rttr: Root TTR (|V| / sqrt(N)) - cttr: Corrected TTR (|V| / sqrt(2N)) - herdan_c: Herdan's C (log |V| / log N) - summer_s: Summer's S (log log |V| / log log N) - dugast: Dugast's U ((log N)^2 / (log N − log |V|)) - maas: Maas a^2 ((log N − log |V|) / (log N)^2) - yule_k: Yule's K (dispersion of frequencies; higher = less diverse) - yule_i: Yule's I (inverse of K, scaled) - herdan_vm: Herdan's Vm - simpson_d: Simpson's D (repeat-probability across tokens) - msttr_{msttr_window}: Mean Segmental TTR over fixed segments - mattr_{mattr_window}: Moving-Average TTR over a sliding window - mtld_{mtld_threshold}: Measure of Textual Lexical Diversity (bidirectional) - hdd_{hdd_draws}: HD-D (expected proportion of types in a sample of size hdd_draws) - vocd_{vocd_ntokens}: VOCD (D) estimated by fitting TTR(N) to a theoretical curve

VOCD estimation. VOCD is fit without external optimization libraries: the function performs a coarse grid search over candidate D values (minimizing squared error between observed mean TTRs and a theoretical TTR(N; D) curve) for multiple repetitions, then averages the best D across repetitions. This generally tracks SciPy-based curve fits closely; you can widen the search grid or add a fine local search if tighter agreement is desired.

Raises:

Type	Description
FileNotFoundError	If analysis_csv is provided but the file does not exist.
ValueError	If none or more than one of csv_path, txt_dir, or analysis_csv are provided, or if the analysis-ready CSV is missing required columns (text_id, text).

Examples:

Analyze an existing analysis-ready CSV (utterance-level):

>>> analyze_lexical_richness(
...     analysis_csv="transcripts_all.csv",
...     out_features_csv="features/lexical-richness.csv",
...     overwrite_existing=True,
... )

Gather from a transcript CSV and aggregate per (source, speaker):

>>> analyze_lexical_richness(
...     csv_path="transcripts/session.csv",
...     text_cols=["text"],
...     id_cols=["source", "speaker"],
...     group_by=["source", "speaker"],
...     mode="concat",
...     out_features_csv="features/lexical-richness.csv",
... )

See Also

analyze_readability : Parallel analyzer producing readability indices.

Source code in src\taters\text\analyze_lexical_richness.py

@records_settings(binding=TEXT_INPUT, grain=TEXT_GRAIN,
                  outputs=("out_features_csv",))
def analyze_lexical_richness(
    *,
    # ----- Input source (exactly one unless analysis_csv is provided) ----------
    csv_path: Optional[Union[str, Path]] = None,
    txt_dir: Optional[Union[str, Path]] = None,
    analysis_csv: Optional[Union[str, Path]] = None,  # if given, we skip gathering
    gathered_csv: Optional[Union[str, Path]] = None,
    on_progress: Optional[Callable[[int, int], None]] = None,

    # ----- Output --------------------------------------------------------------
    out_features_csv: Optional[Union[str, Path]] = None,
    overwrite_existing: bool = False,
    workers: int = 0,

    # ====== SHARED I/O OPTIONS ======
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[Union[str, Path]] = None,

    # ====== TXT FOLDER GATHER OPTIONS ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ====== Metric hyperparameters (optional) ======
    msttr_window: int = 100,
    mattr_window: int = 100,
    mtld_threshold: float = 0.72,
    hdd_draws: int = 42,
    vocd_ntokens: int = 50,
    vocd_within_sample: int = 100,
    vocd_iterations: int = 3,
    vocd_seed: int = 42,

    # ====== NEW: passthrough control ======
    pass_through_cols: Optional[Sequence[str]] = None,
) -> Path:
    """
    Compute lexical richness/diversity metrics for each text row and write a features CSV.
    Draws heavily from https://github.com/LSYS/lexicalrichness but makes several key changes 
    with the goals of minimizing dependencies, attempting to make some speed optimizations with 
    grid search instead of precise curve specifications, and making some principled decisions 
    around punctuation/hyphenization that differ from the original Note that these decisions are 
    not objectively "better" than the original but, instead, reflect my own experiences/intuitions 
    about what makes sense.

    This function accepts (a) an *analysis-ready* CSV (with columns `text_id,text`), (b) a
    raw CSV plus instructions for gathering/aggregation, or (c) a folder of `.txt` files.
    For each resulting row of text, it tokenizes words and computes a suite of classical
    lexical richness measures (e.g., TTR, Herdan's C, Yule's K, MTLD, MATTR, HDD, VOCD).
    Results are written as a wide CSV whose rows align with the rows in the analysis-ready
    table (or the gathered `group_by` rows), preserving any non-text metadata columns.

    Parameters
    ----------
    csv_path : str or Path, optional
        Source CSV to *gather* from. Use with `text_cols`, optional `id_cols`, and
        optional `group_by`. Exactly one of `csv_path`, `txt_dir`, or `analysis_csv`
        must be provided (unless `analysis_csv` is given, which skips gathering).
    txt_dir : str or Path, optional
        Folder of `.txt` files to gather. File identifiers are created from filenames
        via `id_from` and (optionally) a `source_path` column when `include_source_path=True`.
    analysis_csv : str or Path, optional
        Existing analysis-ready CSV with columns `text_id,text`. When provided, all
        gathering options are ignored and the file is used as-is.
    gathered_csv : str or pathlib.Path, optional
        Where to write the intermediate "analysis-ready" table built from
        ``csv_path`` or ``txt_dir``.

        By default it lands beside the *source* -- which means analyzing a
        spreadsheet in someone's Downloads folder writes a file into their
        Downloads folder. Pass this to keep the intermediate with the rest of a
        run's output instead. Ignored when ``analysis_csv`` is given, because
        then no gathering happens.
    on_progress : callable, optional
        Called as ``on_progress(done, total, message=None)`` so a UI can show a
        real bar instead of a spinner.

        ``total`` is the row count of the analysis-ready table, which is fully
        written before measuring starts, so it is known up front. Until it is,
        ``total`` is ``None`` and ``done`` is a running tally -- the long silent
        passes (reading the input, counting its rows) report through the same
        callback with a ``message`` saying which one is running.

        Injected automatically by the pipeline runner for any step function that
        declares this parameter.
    out_features_csv : str or Path, optional
        Output CSV path. If omitted, defaults to
        `./features/lexical-richness/<analysis_ready_filename>`.
    overwrite_existing : bool, default False
        If `False` and `out_features_csv` exists, the function short-circuits and
        returns the existing path without recomputation. This also controls the
        intermediate analysis-ready CSV: when `True`, it is rebuilt from the current
        source instead of reusing a stale copy from an earlier run.
    encoding : str, default "utf-8-sig"
        Encoding for reading/writing CSVs.
    text_cols : sequence of str, default ("text",)
        Text column(s) to use when `csv_path` is provided. When multiple columns are
        given, they are combined according to `mode` (`concat` or `separate`).
    id_cols : sequence of str, optional
        Columns to carry through unchanged into the analysis-ready CSV prior to analysis
        (e.g., `["source","speaker"]`). These will also appear in the output features CSV.
    mode : {"concat", "separate"}, default "concat"
        Gathering behavior when multiple `text_cols` are provided. `"concat"` joins
        values using `joiner`; `"separate"` produces separate rows per text column.
    group_by : sequence of str, optional
        If provided, texts are grouped by these columns before analysis (e.g.,
        `["source","speaker"]`). With `mode="concat"`, all texts in a group are joined
        into one blob per group; with `mode="separate"`, they remain separate rows.
    delimiter : str, default ","
        Column separator of the *input* spreadsheet. The gathered table and
        the output are always comma-separated.
    joiner : str, default " "
        String used to join text fields when `mode="concat"`.
    num_buckets : int, default 512
        Internal streaming/gather parameter to control temporary file bucketing
        (passed through to the gatherer).
    max_open_bucket_files : int, default 64
        Maximum number of temporary files simultaneously open during gathering.
    tmp_root : str or Path, optional
        Temporary directory root for the gatherer. Defaults to a system temp location.
    recursive : bool, default True
        When `txt_dir` is provided, whether to search subdirectories for `.txt` files.
    pattern : str, default "*.txt"
        Glob pattern for discovering text files under `txt_dir`.
    id_from : {"stem", "name", "path"}, default "stem"
        How to construct `text_id` for `.txt` inputs: file stem, full name, or relative path.
    include_source_path : bool, default True
        When `txt_dir` is used, include a `source_path` column in the analysis-ready CSV.
    msttr_window : int, default 100
        Window size for MSTTR (Mean Segmental TTR). Must be smaller than the number of tokens
        in the text to produce a value.
    mattr_window : int, default 100
        Window size for MATTR (Moving-Average TTR). Must be smaller than the number of tokens.
    mtld_threshold : float, default 0.72
        MTLD threshold for factor completion. A higher threshold yields shorter factors and
        typically lower MTLD values; the default follows common practice.
    hdd_draws : int, default 42
        Sample size `n` for HD-D (Hypergeometric Distribution Diversity). Must be less than
        the number of tokens to produce a value.
    vocd_ntokens : int, default 50
        Maximum sample size used to estimate VOCD (D). For each `N` in 35..`vocd_ntokens`,
        the function computes the average TTR over many random samples (`vocd_within_sample`).
    vocd_within_sample : int, default 100
        Number of random samples drawn per `N` when estimating VOCD.
    vocd_iterations : int, default 3
        Repeat-estimate count for VOCD. The best-fit D from each repetition is averaged.
    vocd_seed : int, default 42
        Seed for the VOCD random sampler (controls reproducibility across runs).

    pass_through_cols : Sequence[str] or None, optional
        Extra input columns to copy into the output beside ``text_id``.
    workers : int, default=0
        Parallel processes for reading documents. ``0`` means automatic:
        three-quarters of the logical cores; ``1`` turns parallelism off. Output files are
        identical whatever the worker count.

    Returns
    -------
    Path
        Path to the written features CSV.

    Output shape
    ------------
    The features CSV starts with::

        text_id, <pass-through columns...>, ttr, rttr, cttr, ...

    Pass-through behavior:

    - If ``pass_through_cols`` is provided, those columns are included in that order.
    - Otherwise, if ``id_cols`` were used during gathering, they are included in that order.
    - Otherwise (backward compatible), *all* non-``text`` columns from the analysis-ready CSV are passed through.

    Metrics emitted per row (``None`` if the text is too short):
    ``ttr, rttr, cttr, herdan_c, summer_s, dugast, maas, yule_k, yule_i, herdan_vm, simpson_d,
    msttr_{msttr_window}, mattr_{mattr_window}, mtld_{mtld_threshold}, hdd_{hdd_draws}, vocd_{vocd_ntokens}``.

    Notes
    -----
    **Tokenization and preprocessing.**
    Texts are lowercased, digits are removed, and punctuation characters are
    replaced with spaces prior to tokenization. As a result, hyphenated forms such
    as `"state-of-the-art"` will be split into separate tokens (`"state"`, `"of"`,
    `"the"`, `"art"`). This choice yields robust behavior across corpora but can
    produce different numeric results than implementations that *remove* hyphens
    (treating `"state-of-the-art"` as a single token). If you require strict parity
    with a hyphen-removal scheme, adapt the internal preprocessing accordingly.

    **Metrics.**
    The following measures are emitted per row (values are `None` when a text is
    too short to support the computation):
    - ``ttr``: Type-Token Ratio (|V| / N)
    - ``rttr``: Root TTR (|V| / sqrt(N))
    - ``cttr``: Corrected TTR (|V| / sqrt(2N))
    - ``herdan_c``: Herdan's C (log |V| / log N)
    - ``summer_s``: Summer's S (log log |V| / log log N)
    - ``dugast``: Dugast's U ((log N)^2 / (log N − log |V|))
    - ``maas``: Maas a^2 ((log N − log |V|) / (log N)^2)
    - ``yule_k``: Yule's K (dispersion of frequencies; higher = less diverse)
    - ``yule_i``: Yule's I (inverse of K, scaled)
    - ``herdan_vm``: Herdan's Vm
    - ``simpson_d``: Simpson's D (repeat-probability across tokens)
    - ``msttr_{msttr_window}``: Mean Segmental TTR over fixed segments
    - ``mattr_{mattr_window}``: Moving-Average TTR over a sliding window
    - ``mtld_{mtld_threshold}``: Measure of Textual Lexical Diversity (bidirectional)
    - ``hdd_{hdd_draws}``: HD-D (expected proportion of types in a sample of size ``hdd_draws``)
    - ``vocd_{vocd_ntokens}``: VOCD (D) estimated by fitting TTR(N) to a theoretical curve

    **VOCD estimation.**
    VOCD is fit without external optimization libraries: the function performs a
    coarse grid search over candidate D values (minimizing squared error between
    observed mean TTRs and a theoretical TTR(N; D) curve) for multiple repetitions,
    then averages the best D across repetitions. This generally tracks SciPy-based
    curve fits closely; you can widen the search grid or add a fine local search
    if tighter agreement is desired.

    Raises
    ------
    FileNotFoundError
        If `analysis_csv` is provided but the file does not exist.
    ValueError
        If none or more than one of `csv_path`, `txt_dir`, or `analysis_csv` are provided,
        or if the analysis-ready CSV is missing required columns (`text_id`, `text`).

    Examples
    --------
    Analyze an existing analysis-ready CSV (utterance-level):

    >>> analyze_lexical_richness(
    ...     analysis_csv="transcripts_all.csv",
    ...     out_features_csv="features/lexical-richness.csv",
    ...     overwrite_existing=True,
    ... )

    Gather from a transcript CSV and aggregate per (source, speaker):

    >>> analyze_lexical_richness(
    ...     csv_path="transcripts/session.csv",
    ...     text_cols=["text"],
    ...     id_cols=["source", "speaker"],
    ...     group_by=["source", "speaker"],
    ...     mode="concat",
    ...     out_features_csv="features/lexical-richness.csv",
    ... )

    See Also
    --------
    analyze_readability : Parallel analyzer producing readability indices.
    """
    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        out_features_csv = Path.cwd() / "features" / "lexical-richness" / analysis_ready.name
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)

    if not overwrite_existing and out_features_csv.is_file():
        print(f"Lexical richness output file already exists; returning existing file: {out_features_csv}")
        return out_features_csv

    # 3) name our metrics
    metrics_fixed = list(METRICS_FIXED)
    m_msttr = f"msttr_{msttr_window}"
    m_mattr = f"mattr_{mattr_window}"
    m_mtld  = f"mtld_{str(mtld_threshold).replace('.', '_')}"
    m_hdd   = f"hdd_{hdd_draws}"
    m_vocd  = f"vocd_{vocd_ntokens}"
    metric_names = metrics_fixed + [m_msttr, m_mattr, m_mtld, m_hdd, m_vocd]

    # 4) figure out the output's shape from the input's header alone
    with analysis_ready.open("r", newline="", encoding=encoding) as fin:
        header_fields = csv.DictReader(fin).fieldnames or []

    if "text_id" not in header_fields or "text" not in header_fields:
        raise ValueError(
            f"Expected columns 'text_id' and 'text' in {analysis_ready}; found {header_fields}"
        )

    # there's one shared rule for what rides along beside text_id -- see
    # resolve_passthrough_columns
    from ..helpers.row_map import resolve_passthrough_columns

    passthrough_cols = resolve_passthrough_columns(
        header_fields, pass_through_cols=pass_through_cols, id_cols=id_cols,
        group_by=group_by, analysis_ready=analysis_ready)
    fieldnames = ["text_id", *passthrough_cols, *metric_names]

    # 5) scoring is CPU-bound per row and the rows don't depend on each other,
    #    so we run it on the shared pooled-row driver. that gives us results
    #    in file order no matter the worker count, plus one live sub-bar per
    #    document in flight
    from ..helpers.parallel_map import pool_workers
    from ..helpers.row_map import map_text_rows

    params = dict(msttr_window=msttr_window, mattr_window=mattr_window,
                  mtld_threshold=mtld_threshold, hdd_draws=hdd_draws,
                  vocd_ntokens=vocd_ntokens,
                  vocd_within_sample=vocd_within_sample,
                  vocd_iterations=vocd_iterations, vocd_seed=vocd_seed)
    with atomic_write(out_features_csv, newline="", encoding=encoding) as fout:
        writer = csv.DictWriter(fout, fieldnames=fieldnames)
        writer.writeheader()
        for row, values in map_text_rows(
                analysis_ready, encoding=encoding,
                workers=lambda n_rows: pool_workers(workers, n_rows),
                message="measuring lexical richness", on_progress=on_progress,
                inline_fn=lambda pair: _richness_of(pair[1], params),
                pool_fn=_richness_in_worker,
                initializer=_init_richness_worker, initargs=(params,)):
            out_row: Dict[str, Any] = {
                "text_id": row.get("text_id"),
                **{k: row.get(k) for k in passthrough_cols},
                **dict(zip(metric_names, values)),
            }
            writer.writerow(out_row)
    return out_features_csv

hdd

hdd(tokens, draws=42)

HD-D (McCarthy & Jarvis): sum over types of (1 - P(X=0)) / draws, where X ~ Hypergeom(N, K, n) with N=len(tokens), K=freq(term), n=draws.

Source code in src\taters\text\analyze_lexical_richness.py

def hdd(tokens: List[str], draws: int = 42) -> Optional[float]:
    """
    HD-D (McCarthy & Jarvis): sum over types of (1 - P(X=0)) / draws,
    where X ~ Hypergeom(N, K, n) with N=len(tokens), K=freq(term), n=draws.
    """
    N = len(tokens)
    if N == 0 or draws <= 0 or draws > N:
        return None
    term_freq = Counter(tokens)
    contribs = []
    for K in term_freq.values():
        p0 = _hypergeom_pmf_zero(N, K, draws)
        contribs.append((1 - p0) / draws)
    return sum(contribs)

vocd

vocd(
    tokens,
    ntokens=50,
    within_sample=100,
    iterations=3,
    seed=42,
)

Estimate D by: - for N in 35..ntokens: * sample 'within_sample' subsets of size N, compute TTR, average - grid search D over a reasonable range to minimize squared error to _ttr_nd - repeat 'iterations' times and average the best D

Source code in src\taters\text\analyze_lexical_richness.py

def vocd(tokens: List[str], ntokens: int = 50, within_sample: int = 100,
         iterations: int = 3, seed: int = 42) -> Optional[float]:
    """
    Estimate D by:
      - for N in 35..ntokens:
          * sample 'within_sample' subsets of size N, compute TTR, average
      - grid search D over a reasonable range to minimize squared error to _ttr_nd
      - repeat 'iterations' times and average the best D
    """
    if len(tokens) <= ntokens or ntokens < 35:
        return None
    rng = random.Random(seed)
    Ds: List[float] = []

    # set up our search grid for D (5..200)
    grid: List[float] = []
    # this covers where typical D values live (10..120) with room to spare
    for d in range(5, 201):
        grid.append(float(d))

    for it in range(iterations):
        x_vals: List[int] = []
        y_means: List[float] = []
        for N in range(35, ntokens + 1):
            ttrs: List[float] = []
            for _ in range(within_sample):
                sample = rng.sample(tokens, k=N)
                ttrs.append(len(set(sample)) / N)
            x_vals.append(N)
            y_means.append(mean(ttrs))

        # now we find the D that minimizes the squared error
        best_D = None
        best_err = float("inf")
        for D in grid:
            err = 0.0
            for N, y in zip(x_vals, y_means):
                yhat = _ttr_nd(N, D)
                diff = (y - yhat)
                err += diff * diff
            if err < best_err:
                best_err = err
                best_D = D
        if best_D is not None:
            Ds.append(best_D)

    return mean(Ds) if Ds else None

taters.text.analyze_readability

analyze_readability

analyze_readability(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    on_progress=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    pass_through_cols=None
)

Compute per-row readability metrics using textstat and write a wide features CSV.

The function supports exactly one of three input modes:

1. analysis_csv — Use a prebuilt file with at least columns text_id and text.
2. csv_path — Gather text from an arbitrary CSV using text_cols (and optional id_cols/group_by) to produce an analysis-ready file.
3. txt_dir — Gather text from a folder of .txt files.

If out_features_csv is omitted, the default output path is ./features/readability/<analysis_ready_filename>. All metrics below are computed for every row. Non-numeric metrics (e.g., text_standard) are retained as strings.

Metrics (columns)

The following metrics are emitted as columns (subject to textstat availability):

• flesch_reading_ease
• smog_index
• flesch_kincaid_grade
• coleman_liau_index
• automated_readability_index
• dale_chall_readability_score
• difficult_words
• linsear_write_formula
• gunning_fog
• text_standard (string label)
• spache_readability (for shorter/children texts; may be None)
• syllable_count (on entire text)
• lexicon_count (word count)
• sentence_count
• char_count
• avg_sentence_length
• avg_syllables_per_word
• avg_letter_per_word

Parameters:

Name	Type	Description	Default
csv_path	str or Path	Source CSV to gather from. Mutually exclusive with txt_dir and analysis_csv.	None
txt_dir	str or Path	Folder containing .txt files to gather from. Mutually exclusive with other modes.	None
analysis_csv	str or Path	Prebuilt analysis-ready CSV with columns text_id and text (additional columns such as source/speaker will be copied through to the output).	None
gathered_csv	str or Path	Where to write the intermediate "analysis-ready" table built from csv_path or txt_dir. By default it lands beside the source -- which means analyzing a spreadsheet in someone's Downloads folder writes a file into their Downloads folder. Pass this to keep the intermediate with the rest of a run's output instead. Ignored when analysis_csv is given, because then no gathering happens.	None
on_progress	callable	Called as on_progress(done, total, message=None) so a UI can show a real bar instead of a spinner. total is the row count of the analysis-ready table, which is fully written before measuring starts, so it is known up front. Until it is, total is None and done is a running tally -- the long silent passes (reading the input, counting its rows) report through the same callback with a message saying which one is running. Injected automatically by the pipeline runner for any step function that declares this parameter.	None
out_features_csv	str or Path	Output file path. If None, defaults to ./features/readability/<analysis_ready_filename>.	None
overwrite_existing	bool	If False and the output file already exists, skip processing and return the path. This also controls the intermediate analysis-ready CSV: when True, it is rebuilt from the current source instead of reusing a stale copy from an earlier run.	False
encoding	str	Text encoding used for reading/writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	When gathering from a CSV, name(s) of the column(s) containing text.	("text",)
id_cols	Sequence[str] or None	Optional ID columns to carry into grouping when gathering from CSV.	None
mode	('concat', 'separate')	Gathering behavior when multiple text columns are provided. "concat" joins them using joiner; "separate" creates one row per column.	"concat"
group_by	Sequence[str] or None	Optional grouping keys used during CSV gathering (e.g., ["speaker"]).	None
delimiter	str	Column separator of the input spreadsheet. The gathered table and the output are always comma-separated.	","
joiner	str	Separator used when concatenating multiple text chunks in "concat" mode.	" "
num_buckets	int	Number of temporary hash buckets used during scalable CSV gathering.	512
max_open_bucket_files	int	Maximum number of bucket files kept open concurrently during gathering.	64
tmp_root	str or Path or None	Root directory for temporary gathering artifacts.	None
recursive	bool	When gathering from a text folder, recurse into subdirectories.	True
pattern	str	Glob pattern for selecting text files when gathering from a folder.	"*.txt"
id_from	('stem', 'name', 'path')	How to derive text_id for gathered .txt files.	"stem"
include_source_path	bool	If True, include the absolute source path as an additional column when gathering from a text folder.	True
pass_through_cols	Sequence[str] or None	Extra input columns to copy into the output beside text_id.	None
workers	int	Parallel processes for reading documents. 0 means automatic: three-quarters of the logical cores; 1 turns parallelism off. Output files are identical whatever the worker count.	0

Returns:

Type	Description
Path	Path to the written features CSV.

Output layout

The output CSV starts with: text_id, ,

Pass-through behavior: - If pass_through_cols is provided, those columns are included immediately after text_id in that order. - Else if id_cols were supplied during gathering, they are included in that order. - Else (backward compatible), all non-text columns in the analysis-ready CSV are copied through (e.g., source, speaker, etc.).

Raises:

Type	Description
FileNotFoundError	If an input is missing.
ValueError	If input modes are misconfigured or required columns are absent.
RuntimeError	If textstat is not installed.

Notes

• All rows are processed; blank or missing text yields benign defaults (metrics may be 0 or None).
• Additional columns present in the analysis-ready CSV (beyond text) are copied through to the output (e.g., source, speaker, group_count), aiding joins/aggregation.

Source code in src\taters\text\analyze_readability.py

@records_settings(binding=TEXT_INPUT, grain=TEXT_GRAIN,
                  outputs=("out_features_csv",),
                  # these are just raw sizes; the indices we compute from them
                  # are the actual measures
                  bookkeeping=("lexicon_count", "sentence_count", "char_count",
                               "syllable_count", "difficult_words"))
def analyze_readability(
    *,
    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[Union[str, Path]] = None,
    txt_dir: Optional[Union[str, Path]] = None,
    analysis_csv: Optional[Union[str, Path]] = None,  # if given, we skip gathering
    gathered_csv: Optional[Union[str, Path]] = None,
    on_progress: Optional[Callable[[int, int], None]] = None,

    # ----- Output -----
    out_features_csv: Optional[Union[str, Path]] = None,
    overwrite_existing: bool = False,
    workers: int = 0,

    # ====== SHARED I/O OPTIONS ======
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS ======
    # these only matter when csv_path is provided
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[Union[str, Path]] = None,

    # ====== TXT FOLDER GATHER OPTIONS ======
    # these only matter when txt_dir is provided
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ====== NEW: passthrough control (optional) ======
    pass_through_cols: Optional[Sequence[str]] = None
    ) -> Path:
    """
    Compute per-row readability metrics using `textstat` and write a wide features CSV.

    The function supports exactly one of three input modes:

    1. ``analysis_csv`` — Use a prebuilt file with at least columns ``text_id`` and ``text``.
    2. ``csv_path`` — Gather text from an arbitrary CSV using ``text_cols`` (and optional
       ``id_cols``/``group_by``) to produce an analysis-ready file.
    3. ``txt_dir`` — Gather text from a folder of ``.txt`` files.

    If ``out_features_csv`` is omitted, the default output path is
    ``./features/readability/<analysis_ready_filename>``. All metrics below are computed
    for every row. Non-numeric metrics (e.g., ``text_standard``) are retained as strings.

    Metrics (columns)
    -----------------
    The following metrics are emitted as columns (subject to `textstat` availability):

    - ``flesch_reading_ease``
    - ``smog_index``
    - ``flesch_kincaid_grade``
    - ``coleman_liau_index``
    - ``automated_readability_index``
    - ``dale_chall_readability_score``
    - ``difficult_words``
    - ``linsear_write_formula``
    - ``gunning_fog``
    - ``text_standard``                 (string label)
    - ``spache_readability``            (for shorter/children texts; may be None)
    - ``syllable_count``                (on entire text)
    - ``lexicon_count``                 (word count)
    - ``sentence_count``
    - ``char_count``
    - ``avg_sentence_length``
    - ``avg_syllables_per_word``
    - ``avg_letter_per_word``

    Parameters
    ----------
    csv_path : str or pathlib.Path, optional
        Source CSV to gather from. Mutually exclusive with ``txt_dir`` and ``analysis_csv``.
    txt_dir : str or pathlib.Path, optional
        Folder containing ``.txt`` files to gather from. Mutually exclusive with other modes.
    analysis_csv : str or pathlib.Path, optional
        Prebuilt analysis-ready CSV with columns ``text_id`` and ``text`` (additional columns
        such as ``source``/``speaker`` will be copied through to the output).
    gathered_csv : str or pathlib.Path, optional
        Where to write the intermediate "analysis-ready" table built from
        ``csv_path`` or ``txt_dir``.

        By default it lands beside the *source* -- which means analyzing a
        spreadsheet in someone's Downloads folder writes a file into their
        Downloads folder. Pass this to keep the intermediate with the rest of a
        run's output instead. Ignored when ``analysis_csv`` is given, because
        then no gathering happens.
    on_progress : callable, optional
        Called as ``on_progress(done, total, message=None)`` so a UI can show a
        real bar instead of a spinner.

        ``total`` is the row count of the analysis-ready table, which is fully
        written before measuring starts, so it is known up front. Until it is,
        ``total`` is ``None`` and ``done`` is a running tally -- the long silent
        passes (reading the input, counting its rows) report through the same
        callback with a ``message`` saying which one is running.

        Injected automatically by the pipeline runner for any step function that
        declares this parameter.
    out_features_csv : str or pathlib.Path, optional
        Output file path. If ``None``, defaults to
        ``./features/readability/<analysis_ready_filename>``.
    overwrite_existing : bool, default=False
        If ``False`` and the output file already exists, skip processing and return the path.
        This also controls the intermediate analysis-ready CSV: when ``True``, it is rebuilt
        from the current source instead of reusing a stale copy from an earlier run.
    encoding : str, default="utf-8-sig"
        Text encoding used for reading/writing CSV files.
    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV, name(s) of the column(s) containing text.
    id_cols : Sequence[str] or None, optional
        Optional ID columns to carry into grouping when gathering from CSV.
    mode : {"concat", "separate"}, default="concat"
        Gathering behavior when multiple text columns are provided. ``"concat"`` joins them
        using ``joiner``; ``"separate"`` creates one row per column.
    group_by : Sequence[str] or None, optional
        Optional grouping keys used during CSV gathering (e.g., ``["speaker"]``).
    delimiter : str, default=","
        Column separator of the *input* spreadsheet. The gathered table and
        the output are always comma-separated.
    joiner : str, default=" "
        Separator used when concatenating multiple text chunks in ``"concat"`` mode.
    num_buckets : int, default=512
        Number of temporary hash buckets used during scalable CSV gathering.
    max_open_bucket_files : int, default=64
        Maximum number of bucket files kept open concurrently during gathering.
    tmp_root : str or pathlib.Path or None, optional
        Root directory for temporary gathering artifacts.
    recursive : bool, default=True
        When gathering from a text folder, recurse into subdirectories.
    pattern : str, default="*.txt"
        Glob pattern for selecting text files when gathering from a folder.
    id_from : {"stem", "name", "path"}, default="stem"
        How to derive ``text_id`` for gathered ``.txt`` files.
    include_source_path : bool, default=True
        If ``True``, include the absolute source path as an additional column when gathering
        from a text folder.

    pass_through_cols : Sequence[str] or None, optional
        Extra input columns to copy into the output beside ``text_id``.
    workers : int, default=0
        Parallel processes for reading documents. ``0`` means automatic:
        three-quarters of the logical cores; ``1`` turns parallelism off. Output files are
        identical whatever the worker count.

    Returns
    -------
    pathlib.Path
        Path to the written features CSV.

    Output layout
    -------------
    The output CSV starts with:
        text_id, <pass-through columns...>, <metrics...>

    Pass-through behavior:
    - If `pass_through_cols` is provided, those columns are included immediately
      after `text_id` in that order.
    - Else if `id_cols` were supplied during gathering, they are included in that order.
    - Else (backward compatible), *all* non-`text` columns in the analysis-ready CSV
      are copied through (e.g., `source`, `speaker`, etc.).

    Raises
    ------
    FileNotFoundError
        If an input is missing.
    ValueError
        If input modes are misconfigured or required columns are absent.
    RuntimeError
        If ``textstat`` is not installed.

    Notes
    -----
    - All rows are processed; blank or missing text yields benign defaults (metrics may be 0 or None).
    - Additional columns present in the analysis-ready CSV (beyond ``text``) are copied through
      to the output (e.g., ``source``, ``speaker``, ``group_count``), aiding joins/aggregation.
    """
    _require_textstat()   # fail early, with a message that says what to do

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        out_features_csv = Path.cwd() / "features" / "readability" / analysis_ready.name
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)

    if not overwrite_existing and out_features_csv.is_file():
        print(f"Readability output file already exists; returning existing file: {out_features_csv}")
        return out_features_csv

    # 3) our list of metrics
    metrics = list(METRICS)

    # 4) figure out the output's shape from the input's header alone. the
    # gatherer writes the analysis-ready table with commas no matter what the
    # source used, so `delimiter` stops here and does NOT get passed on. when
    # we did pass it on, a ";" spreadsheet blew up with "Expected columns
    # 'text_id' and 'text' ... found ['text_id,id,text']" -- on the one step
    # that almost every study picks
    with analysis_ready.open("r", newline="", encoding=encoding) as fin:
        header_fields = csv.DictReader(fin).fieldnames or []

    if "text_id" not in header_fields or "text" not in header_fields:
        raise ValueError(
            f"Expected columns 'text_id' and 'text' in {analysis_ready}; found {header_fields}"
        )

    # there's one shared rule for what rides along beside text_id -- see
    # resolve_passthrough_columns for the order of preference and for the two
    # columns it never carries
    from ..helpers.row_map import resolve_passthrough_columns

    passthrough_cols = resolve_passthrough_columns(
        header_fields, pass_through_cols=pass_through_cols, id_cols=id_cols,
        group_by=group_by, analysis_ready=analysis_ready)
    fieldnames = ["text_id", *passthrough_cols, *metrics]

    # 5) scoring is CPU-bound per row and the rows don't depend on each other,
    # so we run it on the shared pooled-row driver. that gives us results in
    # file order no matter the worker count, plus one live sub-bar per
    # document in flight. `_score_text` is the one scorer that both the inline
    # path and the workers run
    from ..helpers.parallel_map import pool_workers
    from ..helpers.row_map import map_text_rows

    with atomic_write(out_features_csv, newline="", encoding=encoding) as fout:
        writer = csv.DictWriter(fout, fieldnames=fieldnames)
        writer.writeheader()
        for row, values in map_text_rows(
                analysis_ready, encoding=encoding,
                workers=lambda n_rows: pool_workers(workers, n_rows),
                message="measuring readability", on_progress=on_progress,
                inline_fn=lambda pair: _score_text(metrics, (pair[1] or "").strip()),
                pool_fn=_readability_in_worker,
                initializer=_init_readability_worker,
                initargs=(list(metrics),)):
            out_row: Dict[str, Any] = {
                "text_id": row.get("text_id"),
                **{k: row.get(k, "") for k in passthrough_cols},
                **dict(zip(metrics, values)),
            }
            writer.writerow(out_row)
    return out_features_csv

taters.text.extract_sentence_embeddings

extract_sentence_embeddings

extract_sentence_embeddings(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    encoding="utf-8-sig",
    delimiter=",",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    model_name="sentence-transformers/all-roberta-large-v1",
    device="auto",
    batch_size=32,
    normalize_l2=True,
    rounding=None,
    show_progress=False,
    on_progress=None,
    pass_through_cols=None,
    verbose=True
)

Average sentence embeddings per row of text and write a wide features CSV.

Supports three mutually exclusive input modes:

1. analysis_csv — Use a prebuilt file with columns text_id and text.
2. csv_path — Gather from a CSV using text_cols (and optional id_cols/group_by) to build an analysis-ready CSV.
3. txt_dir — Gather from a folder of .txt files.

For each row, the text is split into sentences (NLTK if available; otherwise a regex fallback). Each sentence is embedded with a Sentence-Transformers model and the vectors are averaged into one row-level embedding. Optionally, vectors are L2-normalized. The output CSV schema is:

text_id[, <pass_through_cols...>], e0, e1, ..., e{D-1}

If out_features_csv is omitted, the default is ./features/sentence-embeddings/<analysis_ready_filename>. When overwrite_existing is False and the output exists, the function returns the existing path without recomputation.

Parameters:

Name	Type	Description	Default
csv_path	str or Path	Source CSV to gather from. Mutually exclusive with txt_dir and analysis_csv.	None
txt_dir	str or Path	Folder of .txt files to gather from. Mutually exclusive with the other modes.	None
analysis_csv	str or Path	Prebuilt analysis-ready CSV containing exactly text_id and text.	None
gathered_csv	str or Path	Where to write the intermediate "analysis-ready" table built from csv_path or txt_dir.	None
workers	int	Parallel processes for reading documents. 0 means automatic: three-quarters of the logical cores; 1 turns parallelism off. Output files are identical whatever the worker count. By default it lands beside the source -- which means analyzing a spreadsheet in someone's Downloads folder writes a file into their Downloads folder. Pass this to keep the intermediate with the rest of a run's output instead. Ignored when analysis_csv is given, because then no gathering happens.	0
out_features_csv	str or Path	Output features CSV path. If None, a default path is derived from the analysis-ready filename under ./features/sentence-embeddings/.	None
overwrite_existing	bool	If False and the output file already exists, skip processing and return it. This also controls the intermediate analysis-ready CSV: when True, it is rebuilt from the current source instead of reusing a stale copy from an earlier run.	False
verbose	bool	Print incidental notices -- which model is loading, which requested pass-through columns the input did not have. The pipeline runner passes False when a live display owns the screen.	True
pass_through_cols	Sequence[str]	Column names from the analysis-ready CSV to copy into the output alongside text_id (e.g., ["source","speaker"]). Any names given in id_cols are always included automatically, even if not listed here. Missing columns are ignored with a warning.	None
encoding	str	CSV I/O encoding.	"utf-8-sig"
delimiter	str	CSV field delimiter.	","
text_cols	Sequence[str]	When gathering from a CSV: column(s) containing text.	("text",)
id_cols	Sequence[str]	When gathering from a CSV: optional ID columns to carry through.	None
mode	('concat', 'separate')	Gathering behavior if multiple text_cols are provided. "concat" joins them with joiner; "separate" creates one row per column.	"concat"
group_by	Sequence[str]	Optional grouping keys used during CSV gathering (e.g., ["speaker"]).	None
joiner	str	Separator used when concatenating text in "concat" mode.	" "
num_buckets	int	Number of temporary hash buckets for scalable gathering.	512
max_open_bucket_files	int	Maximum number of bucket files kept open concurrently during gathering.	64
tmp_root	str or Path	Root directory for temporary gathering artifacts.	None
recursive	bool	When gathering from a text folder, recurse into subdirectories.	True
pattern	str	Glob pattern for selecting text files.	"*.txt"
id_from	('stem', 'name', 'path')	How to derive text_id when gathering from a text folder.	"stem"
include_source_path	bool	Whether to include the absolute source path as an additional column when gathering from a text folder.	True
model_name	str	Sentence-Transformers model name or path.	"sentence-transformers/all-roberta-large-v1"
device	('auto', 'cuda', 'cpu')	Where to run the embedding model. "auto" uses the GPU when torch reports one that works and falls back to the CPU when it does not; "cuda" insists and raises if it cannot; "cpu" never touches the GPU. Previously there was no way to ask: sentence-transformers takes the GPU whenever torch reports one, which is fine until it is the third model in a pipeline to do so.	"auto"
batch_size	int	Batch size for model encoding.	32
normalize_l2	bool	If True, L2-normalize each row's final vector.	True
rounding	int or None	If provided, round floats to this many decimals (useful for smaller files).	None
show_progress	bool	Print the model's own encoding progress bar. Suppressed whenever on_progress is given, because two bars fighting over the same lines is worse than either alone.	False
on_progress	callable	Called as on_progress(done, total, message=None) so a UI can show a real bar instead of a spinner. Injected automatically by the pipeline runner for any step function that declares this parameter. See :mod:taters.helpers.progress for the contract.	None

Returns:

Type	Description
Path	Path to the written features CSV.

Raises:

Type	Description
FileNotFoundError	If an input file or directory does not exist.
ImportError	If sentence-transformers is not installed.
ValueError	If input modes are misconfigured (e.g., multiple or none provided), or if the analysis-ready CSV lacks text_id/text.

Examples:

Compute row-level embeddings from a transcript CSV, grouped by speaker:

>>> analyze_with_sentence_embeddings(
...     csv_path="transcripts/session.csv",
...     text_cols=["text"], id_cols=["speaker"], group_by=["speaker"],
...     model_name="sentence-transformers/all-roberta-large-v1",
...     normalize_l2=True
... )
PosixPath('.../features/sentence-embeddings/session.csv')

Notes

• Rows with no recoverable sentences produce empty feature cells (not zeros).
• The embedding dimensionality D is taken from the model and used to construct header columns e0..e{D-1}.

Source code in src\taters\text\extract_sentence_embeddings.py

@records_settings(binding=TEXT_INPUT, grain=TEXT_GRAIN,
                  outputs=("out_features_csv",))
def extract_sentence_embeddings(
    *,
    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[Union[str, Path]] = None,
    txt_dir: Optional[Union[str, Path]] = None,
    analysis_csv: Optional[Union[str, Path]] = None,
    gathered_csv: Optional[Union[str, Path]] = None,

    # ----- Output -----
    out_features_csv: Optional[Union[str, Path]] = None,
    overwrite_existing: bool = False,
    workers: int = 0,

    # ====== SHARED I/O OPTIONS ======
    encoding: str = "utf-8-sig",
    delimiter: str = ",",

    # ====== CSV GATHER OPTIONS (when csv_path is provided) ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[Union[str, Path]] = None,

    # ====== TXT FOLDER GATHER OPTIONS (when txt_dir is provided) ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ====== SentenceTransformer options ======
    model_name: str = "sentence-transformers/all-roberta-large-v1",
    device: Optional[str] = "auto",
    batch_size: int = 32,
    normalize_l2: bool = True,       # set True if you want unit-length vectors
    rounding: Optional[int] = None,   # None = full precision; 6 is about float32
    show_progress: bool = False,
    on_progress: Optional[Callable[[int, int], None]] = None,
    pass_through_cols: Optional[Sequence[str]] = None,
    verbose: bool = True,
) -> Path:
    """
    Average sentence embeddings per row of text and write a wide features CSV.

    Supports three mutually exclusive input modes:

    1. ``analysis_csv`` — Use a prebuilt file with columns ``text_id`` and ``text``.
    2. ``csv_path`` — Gather from a CSV using ``text_cols`` (and optional
    ``id_cols``/``group_by``) to build an analysis-ready CSV.
    3. ``txt_dir`` — Gather from a folder of ``.txt`` files.

    For each row, the text is split into sentences (NLTK if available; otherwise
    a regex fallback). Each sentence is embedded with a Sentence-Transformers
    model and the vectors are averaged into one row-level embedding. Optionally,
    vectors are L2-normalized. The output CSV schema is:

    ``text_id[, <pass_through_cols...>], e0, e1, ..., e{D-1}``

    If ``out_features_csv`` is omitted, the default is
    ``./features/sentence-embeddings/<analysis_ready_filename>``. When
    ``overwrite_existing`` is ``False`` and the output exists, the function
    returns the existing path without recomputation.

    Parameters
    ----------
    csv_path : str or pathlib.Path, optional
        Source CSV to gather from. Mutually exclusive with ``txt_dir`` and ``analysis_csv``.
    txt_dir : str or pathlib.Path, optional
        Folder of ``.txt`` files to gather from. Mutually exclusive with the other modes.
    analysis_csv : str or pathlib.Path, optional
        Prebuilt analysis-ready CSV containing exactly ``text_id`` and ``text``.
    gathered_csv : str or pathlib.Path, optional
        Where to write the intermediate "analysis-ready" table built from
        ``csv_path`` or ``txt_dir``.
    workers : int, default=0
        Parallel processes for reading documents. ``0`` means automatic: three-quarters of the
        logical cores; ``1`` turns parallelism off. Output files are
        identical whatever the worker count.

        By default it lands beside the *source* -- which means analyzing a
        spreadsheet in someone's Downloads folder writes a file into their
        Downloads folder. Pass this to keep the intermediate with the rest of a
        run's output instead. Ignored when ``analysis_csv`` is given, because
        then no gathering happens.
    out_features_csv : str or pathlib.Path, optional
        Output features CSV path. If ``None``, a default path is derived from the
        analysis-ready filename under ``./features/sentence-embeddings/``.
    overwrite_existing : bool, default=False
        If ``False`` and the output file already exists, skip processing and return it.
        This also controls the intermediate analysis-ready CSV: when ``True``, it is rebuilt
        from the current source instead of reusing a stale copy from an earlier run.
    verbose : bool, default True
        Print incidental notices -- which model is loading, which requested
        pass-through columns the input did not have. The pipeline runner passes
        False when a live display owns the screen.
    pass_through_cols : Sequence[str], optional
        Column names from the analysis-ready CSV to copy into the output
        alongside ``text_id`` (e.g., ``["source","speaker"]``). **Any names
        given in ``id_cols`` are always included automatically**, even if not
        listed here. Missing columns are ignored with a warning.


    encoding : str, default="utf-8-sig"
        CSV I/O encoding.
    delimiter : str, default=","
        CSV field delimiter.

    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV: column(s) containing text.
    id_cols : Sequence[str], optional
        When gathering from a CSV: optional ID columns to carry through.
    mode : {"concat", "separate"}, default="concat"
        Gathering behavior if multiple ``text_cols`` are provided. ``"concat"`` joins
        them with ``joiner``; ``"separate"`` creates one row per column.
    group_by : Sequence[str], optional
        Optional grouping keys used during CSV gathering (e.g., ``["speaker"]``).
    joiner : str, default=" "
        Separator used when concatenating text in ``"concat"`` mode.
    num_buckets : int, default=512
        Number of temporary hash buckets for scalable gathering.
    max_open_bucket_files : int, default=64
        Maximum number of bucket files kept open concurrently during gathering.
    tmp_root : str or pathlib.Path, optional
        Root directory for temporary gathering artifacts.

    recursive : bool, default=True
        When gathering from a text folder, recurse into subdirectories.
    pattern : str, default="*.txt"
        Glob pattern for selecting text files.
    id_from : {"stem", "name", "path"}, default="stem"
        How to derive ``text_id`` when gathering from a text folder.
    include_source_path : bool, default=True
        Whether to include the absolute source path as an additional column when
        gathering from a text folder.

    model_name : str, default="sentence-transformers/all-roberta-large-v1"
        Sentence-Transformers model name or path.
    device : {"auto", "cuda", "cpu"} | None, default "auto"
        Where to run the embedding model. "auto" uses the GPU when torch reports
        one that works and falls back to the CPU when it does not; "cuda"
        insists and raises if it cannot; "cpu" never touches the GPU. Previously
        there was no way to ask: sentence-transformers takes the GPU whenever
        torch reports one, which is fine until it is the third model in a
        pipeline to do so.
    batch_size : int, default=32
        Batch size for model encoding.
    normalize_l2 : bool, default=True
        If ``True``, L2-normalize each row's final vector.
    rounding : int or None, default=None
        If provided, round floats to this many decimals (useful for smaller files).
    show_progress : bool, default=False
        Print the model's own encoding progress bar. Suppressed whenever
        ``on_progress`` is given, because two bars fighting over the same
        lines is worse than either alone.
    on_progress : callable, optional
        Called as ``on_progress(done, total, message=None)`` so a UI can show a
        real bar instead of a spinner. Injected automatically by the pipeline
        runner for any step function that declares this parameter. See
        :mod:`taters.helpers.progress` for the contract.

    Returns
    -------
    pathlib.Path
        Path to the written features CSV.

    Raises
    ------
    FileNotFoundError
        If an input file or directory does not exist.
    ImportError
        If ``sentence-transformers`` is not installed.
    ValueError
        If input modes are misconfigured (e.g., multiple or none provided),
        or if the analysis-ready CSV lacks ``text_id``/``text``.

    Examples
    --------
    Compute row-level embeddings from a transcript CSV, grouped by speaker:

    >>> analyze_with_sentence_embeddings(
    ...     csv_path="transcripts/session.csv",
    ...     text_cols=["text"], id_cols=["speaker"], group_by=["speaker"],
    ...     model_name="sentence-transformers/all-roberta-large-v1",
    ...     normalize_l2=True
    ... )
    PosixPath('.../features/sentence-embeddings/session.csv')

    Notes
    -----
    - Rows with no recoverable sentences produce **empty** feature cells (not zeros).
    - The embedding dimensionality ``D`` is taken from the model and used to
    construct header columns ``e0..e{D-1}``.
    """

    def _merge_cols(preferred: Optional[Sequence[str]], ensure: Optional[Sequence[str]]) -> list[str]:
        """
        Merge two sequences while preserving order and removing duplicates.
        'preferred' order is kept; any 'ensure' items not present are appended.
        """
        out: list[str] = []
        seen: set[str] = set()
        for seq in (preferred or []), (ensure or []):
            for c in seq:
                if c not in seen:
                    out.append(c)
                    seen.add(c)
        return out

    # pre-check that nltk's sent_tokenizer is usable. we call this purely for
    # the side effect (it downloads `punkt` if it's missing), and the sentence
    # splitter falls back on its own if it's still not there, so there's no
    # return value worth keeping
    _ensure_nltk_punkt(verbose=verbose)

    # we resolve these BEFORE the gather, because the gather is what has to
    # preserve them. `pass_through_cols` used to only get applied when reading
    # the analysis-ready CSV back, but the gather that produced it wrote
    # `text_id` and `text` and nothing else. so every requested column came
    # out present-but-empty, and anything grouping on them downstream silently
    # collapsed into one meaningless bucket
    pt_cols: list[str] = _merge_cols(pass_through_cols, id_cols)

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers,
        carry_cols=pt_cols or None, verbose=verbose)

    if out_features_csv is None:
        out_features_csv = Path.cwd() / "features" / "sentence-embeddings" / analysis_ready.name
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)

    if not overwrite_existing and Path(out_features_csv).is_file():
        if verbose:
            print("Sentence embedding feature output file already exists; returning existing file.")
        return out_features_csv

    # 2) load model
    #
    # we import this here rather than at module scope. it used to be guarded up
    # there, which keeps a missing install from breaking the import -- but
    # that's not lazy, and sentence-transformers takes about fourteen seconds
    # to load. the wizard imports this module just to read its signature when
    # it builds the options screen, and was paying that every single time
    try:
        from sentence_transformers import SentenceTransformer
    except Exception as e:
        raise ImportError(
            "sentence-transformers is required. Install with `pip install sentence-transformers`."
        ) from e
    # we resolve the device ourselves rather than leaving it to
    # sentence-transformers, which grabs CUDA whenever torch reports it and
    # gives us no way to find out afterwards what it picked. a step that
    # silently moves to the GPU is a step that can silently run it out of memory
    resolved, fallback_reason = resolve_device(device, backend="torch")
    if verbose:
        print(f"Loading sentence-transformer model: {model_name} on {resolved}")
        if fallback_reason:
            print(f"[sentence-embeddings] {fallback_reason}")
    # and on the run display too, not only under `verbose` -- see the same
    # announcement in transformer_embeddings for why
    announce(on_progress, device_note("embedding", resolved, fallback_reason))
    model = SentenceTransformer(model_name, device=resolved)
    dim = int(getattr(model, "get_sentence_embedding_dimension", lambda: 768)())

    # 3) header
    header = ["text_id"] + pt_cols + [f"e{i}" for i in range(dim)]


    # 4) stream rows → split → encode → average → (optional) L2 normalize → write
    if verbose:
        print("Extracting embeddings...")
    with atomic_write(out_features_csv, newline="", encoding=encoding) as f:
        writer = csv.writer(f)
        writer.writerow(header)

        # open the analysis-ready CSV as dicts so we can read the extra cols
        with analysis_ready.open("r", newline="", encoding=encoding) as rf:
            reader = csv.DictReader(rf, delimiter=delimiter)
            # light validation: warn if any requested pass-through column is missing
            missing = [c for c in pt_cols if c not in (reader.fieldnames or [])]
            if missing and verbose:
                print(f"[sentence-embeddings] WARNING: pass-through columns missing in source: {missing}")

            _ticker = Ticker(on_progress,
                             count_rows(analysis_ready, on_progress=on_progress))

            for row in reader:
                text_id = str(row.get("text_id", ""))
                text = (row.get("text") or "")
                pt_vals = [row.get(c, "") for c in pt_cols]

                sents = _split_sentences(text)
                if not sents:
                    vec = None
                else:
                    emb = model.encode(
                        sents,
                        batch_size=batch_size,
                        convert_to_numpy=True,
                        normalize_embeddings=False,
                        show_progress_bar=show_progress and on_progress is None,
                    )
                    vec = emb.mean(axis=0).astype(np.float32, copy=False)

                if vec is None:
                    values = [""] * dim
                else:
                    if normalize_l2:
                        n = float(np.linalg.norm(vec))
                        if n > 1e-12:
                            vec = vec / n
                    values = [float(x) for x in vec.tolist()]
                    if rounding is not None:
                        values = [round(v, int(rounding)) for v in values]

                writer.writerow([text_id] + pt_vals + values)
                _ticker.tick()


    return out_features_csv

taters.text.finetune_predictor

Fine-tune a transformer to predict outcomes from text -- one outcome or several at once -- with the same cross-validated honesty as the ridge.

A ridge over features asks "which measures predict this outcome?"; a fine-tuned encoder asks "how well can the text itself predict it?", with the encoder's own weights adjusted to the task. Every outcome gets a head of its own on one shared encoder: a regression head (mean squared error on the standardized outcome) for a numeric column, a classification head (cross-entropy) for a categorical one, and a row missing one outcome still trains the others -- so a sheet with a personality score and a diagnosis column trains one model for both (multi-task learning, Caruana 1997), which is the usual way to get more out of a small corpus.

• Devlin, J., et al. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL 2019.
• Caruana, R. (1997). Multitask learning. Machine Learning, 28, 41–75.

The discipline is the ridge's: every headline number is out of fold. The rows are dealt into folds (balanced on the first outcome, stratified when it is categorical); each fold trains a fresh model on the rest, with a slice of the training rows held out for early stopping, and predicts the rows it never saw; the metrics are computed over those predictions, per fold for the standard error and pooled for the headline. A final model is then trained on every row for the median best epoch and saved -- encoder and heads -- as a model any pipeline can apply to new text, whose class predictions carry the data's own labels.

Predicted classes are written as labels, never indices, and every label can be renamed per model in Settings (class_names); so can the settings that govern how the model is applied (apply).

apply_text_predictor

apply_text_predictor(
    *,
    model_json,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    on_progress=None,
    verbose=True,
    encoding="utf-8-sig",
    delimiter=",",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    device="auto",
    rounding=4,
    batch_size=None,
    max_length=None,
    precision=None,
    emit_probabilities=None
)

Score new texts with a fine-tuned text predictor.

The model's own apply settings (batch size, max_length, precision, whether to write the per-class probabilities) are used unless the call gives its own. Predicted classes are written as the model's labels -- the data's own, or whatever they were renamed to in Settings.

Parameters:

Name	Type	Description	Default
model_json	PathLike	A predictor manifest written by :func:finetune_text_predictor; its weights are read from beside it, or from the library when the manifest traveled alone.	required
csv_path	Optional[PathLike]	The same input contract as the other text analyzers.	None
txt_dir	Optional[PathLike]	The same input contract as the other text analyzers.	None
analysis_csv	Optional[PathLike]	The same input contract as the other text analyzers.	None
gathered_csv	Optional[PathLike]	The same input contract as the other text analyzers.	None
out_features_csv	str or Path	Default ./features/text_predictor_applied.csv.	None
overwrite_existing	bool	If False and the output exists, return it untouched.	False
workers	int	Parallel processes for the gather and CPU threads for torch.	0
text_cols	sequence of str	When gathering from a CSV, the column(s) holding the text.	("text",)
id_cols	sequence of str	Columns that identify each row when gathering from a CSV.	None
mode	('concat', 'separate')	With several text columns: join or treat separately.	"concat"
group_by	sequence of str	Columns to combine rows by before scoring.	None
pattern	str	Which files to read from a folder of documents.	every document type
device	('auto', 'cuda', 'cpu')	Where the model runs.	"auto"
rounding	int	Decimal places written.	4
batch_size	Optional[int]	Overrides for the model's own apply settings.	None
max_length	Optional[int]	Overrides for the model's own apply settings.	None
precision	Optional[int]	Overrides for the model's own apply settings.	None
emit_probabilities	Optional[int]	Overrides for the model's own apply settings.	None

Returns:

Type	Description
Path	out_features_csv: text_id, token_count, n_windows, then per outcome pred_<label> and, for a category, prob_<label> and p_<label>_<class> columns. A text longer than max_length is scored in windows and gets the mean of their predictions.

Source code in src\taters\text\finetune_predictor.py

@records_settings(binding=TEXT_INPUT, grain=TEXT_GRAIN, assets={"model_json": None},
                  outputs=("out_features_csv",), bookkeeping=("token_count", "n_windows"))
def apply_text_predictor(
    *,
    model_json: PathLike,
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    out_features_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    workers: int = 0,
    on_progress: Optional[Callable[..., None]] = None,
    verbose: bool = True,
    encoding: str = "utf-8-sig",
    delimiter: str = ",",
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,
    device: Literal["auto", "cuda", "cpu"] = "auto",
    rounding: int = 4,
    batch_size: Optional[int] = None,
    max_length: Optional[int] = None,
    precision: Optional[Literal["auto", "fp32", "fp16"]] = None,
    emit_probabilities: Optional[bool] = None,
) -> Path:
    """
    Score new texts with a fine-tuned text predictor.

    The model's own ``apply`` settings (batch size, max_length, precision,
    whether to write the per-class probabilities) are used unless the call
    gives its own. Predicted classes are written as the model's labels --
    the data's own, or whatever they were renamed to in Settings.

    Parameters
    ----------
    model_json
        A predictor manifest written by :func:`finetune_text_predictor`;
        its weights are read from beside it, or from the library when the
        manifest traveled alone.
    csv_path, txt_dir, analysis_csv, gathered_csv, ...
        The same input contract as the other text analyzers.
    out_features_csv : str or Path, optional
        Default ``./features/text_predictor_applied.csv``.
    overwrite_existing : bool, default False
        If False and the output exists, return it untouched.
    workers : int, default 0
        Parallel processes for the gather and CPU threads for torch.
    text_cols : sequence of str, default ("text",)
        When gathering from a CSV, the column(s) holding the text.
    id_cols : sequence of str, optional
        Columns that identify each row when gathering from a CSV.
    mode : {"concat", "separate"}, default "concat"
        With several text columns: join or treat separately.
    group_by : sequence of str, optional
        Columns to combine rows by before scoring.
    pattern : str, default every document type
        Which files to read from a folder of documents.
    device : {"auto", "cuda", "cpu"}, default "auto"
        Where the model runs.
    rounding : int, default 4
        Decimal places written.
    batch_size, max_length, precision, emit_probabilities
        Overrides for the model's own apply settings.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id, token_count, n_windows,`` then per
        outcome ``pred_<label>`` and, for a category, ``prob_<label>`` and
        ``p_<label>_<class>`` columns. A text longer than ``max_length`` is
        scored in windows and gets the mean of their predictions.
    """
    from ..helpers.model_spec import apply_defaults

    loaded = _load_model(one_model_path(model_json))
    doc, payload = loaded["doc"], loaded["payload"]
    defaults = apply_defaults(doc)
    batch_size = int(batch_size if batch_size is not None else defaults.get("batch_size", 32))
    max_length = int(max_length if max_length is not None
                     else defaults.get("max_length", doc["encoder"]["max_length"]))
    precision = str(precision if precision is not None else defaults.get("precision", "auto"))
    emit = bool(emit_probabilities if emit_probabilities is not None
                else defaults.get("emit_probabilities", True))
    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)
    out_features_csv = Path(out_features_csv) if out_features_csv else \
        Path.cwd() / "features" / "text_predictor_applied.csv"
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)
    if not overwrite_existing and out_features_csv.is_file():
        if verbose:
            print("Text predictor output already exists; returning existing file.")
        return out_features_csv

    import numpy as np

    set_threads(workers)
    announce(on_progress, f"loading {doc.get('name')}")
    model, tokenizer, _res, device_name, _reason = load_encoder(
        payload / "encoder", device=device, verbose=verbose)
    tasks = {name: dict(spec) for name, spec in doc["outcomes"].items()}
    n_layers = int(getattr(model.config, "num_hidden_layers", 0) or
                   getattr(model.config, "n_layers", 0))
    layer_idx, combine = parse_layers(doc["encoder"]["layers"], n_layers)
    hidden = int(model.config.hidden_size) * (len(layer_idx) if combine == "concat" else 1)
    heads = _build_heads(hidden, tasks)
    import torch

    with torch.no_grad():
        for name in tasks:
            key = tasks[name].get("head") or _key(name)
            heads[_key(name)][1].weight.copy_(loaded["heads"][f"{key}.weight"])
            heads[_key(name)][1].bias.copy_(loaded["heads"][f"{key}.bias"])
    heads.to(device_name)

    with analysis_ready.open("r", newline="", encoding=encoding) as fh:
        rows = list(csv.DictReader(fh, delimiter=delimiter))
    texts = [r.get("text") or "" for r in rows]
    announce(on_progress, f"scoring {len(rows)} text(s)")
    # the encoder's own ceiling wins over the setting, and says so rather than
    # quietly reading less than was asked for
    max_length, capped = window_length(tokenizer, max_length)
    if capped:
        announce(on_progress, capped)
    preds, counts, n_windows = _predict(
        model, heads, tokenizer, texts, tasks, layer_idx=layer_idx, combine=combine,
        pooling=doc["encoder"]["pooling"], max_length=max_length, batch_size=batch_size,
        device_name=device_name, precision=precision, verbose=verbose)
    header = ["text_id", "token_count", "n_windows"]
    for name, spec in tasks.items():
        label = output_label(doc, name)
        header.append(f"pred_{label}")
        if spec["task"] == "classification" and emit:
            header += [f"prob_{label}"] + [f"p_{label}_{class_label(doc, name, c)}"
                                           for c in spec["classes"]]
    with atomic_write(out_features_csv, mode="w", newline="", encoding=encoding) as out:
        writer = csv.writer(out)
        writer.writerow(header)
        for i, r in enumerate(rows):
            blank = not texts[i].strip()
            cells: list = [r.get("text_id", ""), counts[i], n_windows[i]]
            for name, spec in tasks.items():
                p = preds[name][i]
                if spec["task"] == "regression":
                    cells.append("" if blank else _fmt(p, rounding))
                else:
                    cls = spec["classes"]
                    cells.append("" if blank else class_label(doc, name, cls[int(np.argmax(p))]))
                    if emit:
                        cells.append("" if blank else _fmt(max(p), rounding))
                        cells += ["" if blank else _fmt(v, rounding) for v in p]
            writer.writerow(cells)
    if verbose:
        print(f"[text_predictor] {len(rows)} text(s) scored with {doc.get('name')} "
              f"on {device_name} -> {out_features_csv}")
    return out_features_csv

finetune_text_predictor

finetune_text_predictor(
    *,
    csv_path=None,
    analysis_csv=None,
    gathered_csv=None,
    out_dir="stats_results",
    out_models_dir=None,
    name=None,
    overwrite_existing=False,
    workers=0,
    on_progress=None,
    verbose=True,
    encoding="utf-8-sig",
    delimiter=",",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    outcome_cols=(),
    categorical_outcomes=(),
    task_weights="",
    base_model=CURATED_ENCODERS[0][0],
    layers="last",
    pooling="mean",
    max_length=256,
    train_layers=0,
    gradient_checkpointing=False,
    n_folds=5,
    stratify=True,
    epochs=3,
    learning_rate=2e-05,
    batch_size=16,
    grad_accum=1,
    weight_decay=0.01,
    warmup_fraction=0.06,
    early_stopping=True,
    val_fraction=0.1,
    device="auto",
    precision="auto",
    seed=42,
    rounding=4
)

Fine-tune an encoder to predict one or more outcome columns from text.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The spreadsheet: the text in text_cols and the outcomes in outcome_cols, one row per text (or a prebuilt analysis-ready table carrying the outcome columns). A folder of documents has no outcome columns and is not accepted.	None
analysis_csv	Optional[PathLike]	The spreadsheet: the text in text_cols and the outcomes in outcome_cols, one row per text (or a prebuilt analysis-ready table carrying the outcome columns). A folder of documents has no outcome columns and is not accepted.	None
gathered_csv	Optional[PathLike]	The spreadsheet: the text in text_cols and the outcomes in outcome_cols, one row per text (or a prebuilt analysis-ready table carrying the outcome columns). A folder of documents has no outcome columns and is not accepted.	None
out_dir	str or Path	Where the metrics, fold, epoch and prediction tables, the report section and the figures go.	"stats_results"
out_models_dir	str or Path	Where the model lands, default <out_dir>/models: a manifest <name>.json (text_predictor__<outcomes>.json when no name is given) and its weights folder beside it.	None
name	str	The model's name in menus and its file stem; default text_predictor__<outcomes>.	None
overwrite_existing	bool	If False and the metrics table exists, return it untouched.	False
workers	int	Parallel processes for the gather, and the CPU threads torch may use. 0 means automatic.	0
text_cols	sequence of str	The column(s) holding the text.	("text",)
id_cols	sequence of str	Columns that identify each row.	None
mode	('concat', 'separate')	With several text columns: join them into one text per row, or treat each as its own text.	"concat"
group_by	sequence of str	Columns to combine rows by before training (one text per group); an outcome then has to be constant within a group.	None
outcome_cols	sequence of str	The columns to predict. A numeric column is a regression (mean squared error on the standardized value); a column of labels, or one named in categorical_outcomes, is a classification. Several columns train one model with a head per outcome (multi-task).	()
categorical_outcomes	sequence of str	Which of outcome_cols are categories even though they look numeric -- a 0/1 condition code.	()
task_weights	str	"age: 1, condition: 2": how much each outcome's loss counts; unsaid means 1.	''
base_model	str	The encoder to start from: a Hugging Face name, a checkpoint folder, a Taters text encoder file, or a Taters fine-tuned predictor file -- in which case its encoder is the starting point and its heads are reused for outcomes with the same name and type (a warm start on new data), with fresh heads for new outcomes.	CURATED_ENCODERS[0][0]
layers	str	Which hidden layers feed the heads (last is right when the encoder is being trained; see the embeddings step for the others).	"last"
pooling	('mean', 'cls', 'max')	How a text's token vectors become one.	"mean"
max_length	int	The most tokens read per text; longer texts are cut (the share cut is reported).	256
train_layers	int	Train only the top this-many encoder layers and the heads; 0 trains everything. Two is a good CPU compromise.	0
gradient_checkpointing	bool	Trade compute for memory on a small card.	False
n_folds	int	Cross-validation folds. Every headline number is out of fold.	5
stratify	bool	Deal the folds balanced on the first outcome (stratified when it is a category) rather than at random.	True
epochs	int	The most passes over the training rows per fold; with early stopping the epoch with the lowest validation loss is kept.	3
learning_rate	float	The peak learning rate of AdamW, after warm-up.	2e-5
batch_size	int	Texts per forward pass; halved automatically if the GPU runs out of memory (accumulation doubled to compensate).	16
grad_accum	int	Batches accumulated per optimizer step.	1
weight_decay	float	AdamW's weight decay; the share of steps spent warming up.	0.01
warmup_fraction	float	AdamW's weight decay; the share of steps spent warming up.	0.01
early_stopping	bool	Keep, per fold, the epoch with the lowest loss on a validation slice of the training rows (val_fraction); the final model trains for the median best epoch.	True
val_fraction	float	The share of each fold's training rows held out for early stopping.	0.1
device	('auto', 'cuda', 'cpu')	Where training runs.	"auto"
precision	('auto', 'fp32', 'fp16')	Half precision on a GPU (auto), always full, or always half.	"auto"
seed	int	Seeds the folds, the shuffles, the validation slices and the heads.	42
rounding	int	Decimal places written.	4

Returns:

Type	Description
Path	<out_dir>/text_predictor_cv_metrics.csv: one row per outcome with the ridge's and the classifier's columns, so the two kinds of model can be compared in one table.

Source code in src\taters\text\finetune_predictor.py

def finetune_text_predictor(
    *,
    csv_path: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    out_dir: PathLike = "stats_results",
    out_models_dir: Optional[PathLike] = None,
    name: Optional[str] = None,
    overwrite_existing: bool = False,
    workers: int = 0,
    on_progress: Optional[Callable[..., None]] = None,
    verbose: bool = True,
    encoding: str = "utf-8-sig",
    delimiter: str = ",",
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ----- what to predict -----
    outcome_cols: Sequence[str] = (),
    categorical_outcomes: Sequence[str] = (),
    task_weights: str = "",

    # ----- the model -----
    base_model: str = CURATED_ENCODERS[0][0],
    layers: str = "last",
    pooling: Literal["mean", "cls", "max"] = "mean",
    max_length: int = 256,
    train_layers: int = 0,
    gradient_checkpointing: bool = False,

    # ----- training -----
    n_folds: int = 5,
    stratify: bool = True,
    epochs: int = 3,
    learning_rate: float = 2e-5,
    batch_size: int = 16,
    grad_accum: int = 1,
    weight_decay: float = 0.01,
    warmup_fraction: float = 0.06,
    early_stopping: bool = True,
    val_fraction: float = 0.1,
    device: Literal["auto", "cuda", "cpu"] = "auto",
    precision: Literal["auto", "fp32", "fp16"] = "auto",
    seed: int = 42,
    rounding: int = 4,
) -> Path:
    """
    Fine-tune an encoder to predict one or more outcome columns from text.

    Parameters
    ----------
    csv_path, analysis_csv, gathered_csv
        The spreadsheet: the text in ``text_cols`` and the outcomes in
        ``outcome_cols``, one row per text (or a prebuilt analysis-ready
        table carrying the outcome columns). A folder of documents has no
        outcome columns and is not accepted.
    out_dir : str or Path, default "stats_results"
        Where the metrics, fold, epoch and prediction tables, the report
        section and the figures go.
    out_models_dir : str or Path, optional
        Where the model lands, default ``<out_dir>/models``: a manifest
        ``<name>.json`` (``text_predictor__<outcomes>.json`` when no name is
        given) and its weights folder beside it.
    name : str, optional
        The model's name in menus and its file stem; default
        ``text_predictor__<outcomes>``.
    overwrite_existing : bool, default False
        If False and the metrics table exists, return it untouched.
    workers : int, default 0
        Parallel processes for the gather, and the CPU threads torch may
        use. 0 means automatic.
    text_cols : sequence of str, default ("text",)
        The column(s) holding the text.
    id_cols : sequence of str, optional
        Columns that identify each row.
    mode : {"concat", "separate"}, default "concat"
        With several text columns: join them into one text per row, or
        treat each as its own text.
    group_by : sequence of str, optional
        Columns to combine rows by before training (one text per group);
        an outcome then has to be constant within a group.
    outcome_cols : sequence of str
        The columns to predict. A numeric column is a regression (mean
        squared error on the standardized value); a column of labels, or
        one named in ``categorical_outcomes``, is a classification. Several
        columns train one model with a head per outcome (multi-task).
    categorical_outcomes : sequence of str
        Which of ``outcome_cols`` are categories even though they look
        numeric -- a 0/1 condition code.
    task_weights : str
        ``"age: 1, condition: 2"``: how much each outcome's loss counts;
        unsaid means 1.
    base_model : str
        The encoder to start from: a Hugging Face name, a checkpoint
        folder, a Taters text encoder file, or a Taters fine-tuned
        predictor file -- in which case its encoder is the starting point
        and its heads are reused for outcomes with the same name and type
        (a warm start on new data), with fresh heads for new outcomes.
    layers : str, default "last"
        Which hidden layers feed the heads (``last`` is right when the
        encoder is being trained; see the embeddings step for the others).
    pooling : {"mean", "cls", "max"}, default "mean"
        How a text's token vectors become one.
    max_length : int, default 256
        The most tokens read per text; longer texts are cut (the share cut
        is reported).
    train_layers : int, default 0
        Train only the top this-many encoder layers and the heads; 0 trains
        everything. Two is a good CPU compromise.
    gradient_checkpointing : bool, default False
        Trade compute for memory on a small card.
    n_folds : int, default 5
        Cross-validation folds. Every headline number is out of fold.
    stratify : bool, default True
        Deal the folds balanced on the first outcome (stratified when it
        is a category) rather than at random.
    epochs : int, default 3
        The most passes over the training rows per fold; with early
        stopping the epoch with the lowest validation loss is kept.
    learning_rate : float, default 2e-5
        The peak learning rate of AdamW, after warm-up.
    batch_size : int, default 16
        Texts per forward pass; halved automatically if the GPU runs out of
        memory (accumulation doubled to compensate).
    grad_accum : int, default 1
        Batches accumulated per optimizer step.
    weight_decay, warmup_fraction
        AdamW's weight decay; the share of steps spent warming up.
    early_stopping : bool, default True
        Keep, per fold, the epoch with the lowest loss on a validation
        slice of the training rows (``val_fraction``); the final model
        trains for the median best epoch.
    val_fraction : float, default 0.1
        The share of each fold's training rows held out for early stopping.
    device : {"auto", "cuda", "cpu"}, default "auto"
        Where training runs.
    precision : {"auto", "fp32", "fp16"}, default "auto"
        Half precision on a GPU (auto), always full, or always half.
    seed : int, default 42
        Seeds the folds, the shuffles, the validation slices and the heads.
    rounding : int, default 4
        Decimal places written.

    Returns
    -------
    Path
        ``<out_dir>/text_predictor_cv_metrics.csv``: one row per outcome
        with the ridge's and the classifier's columns, so the two kinds of
        model can be compared in one table.
    """
    missing = torch_missing_reason()
    if missing:
        raise ImportError(missing)
    outcome_cols = [str(c) for c in outcome_cols if str(c).strip()]
    if not outcome_cols:
        raise ValueError("outcome_cols names no column to predict")
    if int(n_folds) < 2:
        raise ValueError("n_folds must be at least 2: every headline number is out of fold")
    if int(epochs) < 1 or int(batch_size) < 1:
        raise ValueError("epochs and batch_size must be at least 1")
    if pooling not in ("mean", "cls", "max"):
        raise ValueError(f"pooling must be mean, cls or max, not {pooling!r}")
    started = time.time()
    # we grab this now, before the per-outcome loops below rebind `name`. we
    # used to read it afterwards, and the model got named after whichever
    # outcome happened to come last
    model_name = str(name).strip() if name else ""
    out_dir = Path(out_dir)
    models_dir = Path(out_models_dir) if out_models_dir else out_dir / "models"
    metrics_path = out_dir / "text_predictor_cv_metrics.csv"
    if metrics_path.is_file() and not overwrite_existing:
        if verbose:
            print("Text predictor results already exist; returning existing file.")
        return metrics_path
    weights = parse_task_weights(task_weights, outcome_cols)

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=None, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers, carry_cols=list(outcome_cols), verbose=verbose)
    with analysis_ready.open("r", newline="", encoding=encoding) as fh:
        table = [r for r in csv.DictReader(fh, delimiter=delimiter)
                 if (r.get("text") or "").strip()]
    missing_cols = [c for c in outcome_cols if table and c not in table[0]]
    if missing_cols:
        raise ValueError(
            f"outcome column(s) {', '.join(missing_cols)} are not in the table "
            f"(it has {', '.join(table[0].keys())})")
    tasks = _outcome_tasks(table, outcome_cols, [str(c) for c in categorical_outcomes])
    labeled = [i for i, r in enumerate(table)
                if any((r.get(c) or "").strip() for c in outcome_cols)]
    if len(labeled) < 2 * int(n_folds):
        raise ValueError(
            f"{len(labeled)} row(s) have an outcome; {n_folds}-fold cross-validation "
            f"needs at least {2 * int(n_folds)}.")
    table = [table[i] for i in labeled]
    texts = [r.get("text") or "" for r in table]
    ids = [r.get("text_id", "") for r in table]

    import numpy as np
    import torch

    threads = set_threads(workers)
    announce(on_progress, f"loading {base_model}")
    warm = _load_warm(str(base_model))
    resolved = resolve_encoder(base_model)
    _model0, tokenizer, _res, device_name, fallback = load_encoder(
        base_model, device=device, verbose=verbose)
    n_layers = int(getattr(_model0.config, "num_hidden_layers", 0) or
                   getattr(_model0.config, "n_layers", 0))
    hidden_size = int(_model0.config.hidden_size)
    n_params = int(sum(p.numel() for p in _model0.parameters()))
    del _model0
    layer_idx, combine = parse_layers(layers, n_layers)
    common = dict(layer_idx=layer_idx, combine=combine, pooling=pooling,
                  max_length=int(max_length), batch_size=int(batch_size),
                  grad_accum=int(grad_accum), epochs=int(epochs),
                  learning_rate=float(learning_rate), weight_decay=float(weight_decay),
                  warmup_fraction=float(warmup_fraction), train_layers=int(train_layers),
                  gradient_checkpointing=bool(gradient_checkpointing),
                  device_name=device_name, precision=precision,
                  early_stopping=bool(early_stopping), on_progress=on_progress,
                  verbose=verbose)
    predict_common = dict(layer_idx=layer_idx, combine=combine, pooling=pooling,
                          max_length=int(max_length), batch_size=int(batch_size),
                          device_name=device_name, precision=precision, verbose=verbose)
    # every text's windows, cut once and shared by every fold and the final
    # model (tokenizing is cheap, but there's no reason to do it six times)
    announce(on_progress, "tokenizing the texts")
    windows, _token_counts = _windows_of(tokenizer, texts, int(max_length))
    common["windows"] = windows

    # first, cross-validation
    folds = _fold_of(table, tasks, outcome_cols[0], int(n_folds), int(seed), bool(stratify))
    oof: Dict[str, list] = {name: [None] * len(table) for name in tasks}
    fold_rows: List[list] = []
    epoch_rows: List[list] = []
    best_epochs: List[int] = []
    reused_heads: List[str] = []
    for f in range(int(n_folds)):
        test_rows = [i for i in range(len(table)) if folds[i] == f]
        rest = [i for i in range(len(table)) if folds[i] != f]
        train_rows, val_rows = _split_validation(rest, val_fraction if early_stopping else 0.0,
                                                 int(seed) + f)
        announce(on_progress, f"fold {f + 1}/{n_folds}: training on {len(train_rows)} texts")
        model, heads, history, best = _train_one(
            resolved.source, tokenizer, table, tasks, weights, train_rows, val_rows,
            seed=int(seed) + f, warm_heads=warm, label=f"fold {f + 1}", **common)
        if warm and f == 0:
            fresh = _build_heads(hidden_size * (len(layer_idx) if combine == "concat" else 1), tasks)
            reused_heads = _copy_heads(fresh, warm, tasks)
        best_epochs.append(best)
        for h in history:
            epoch_rows.append([f + 1, h["epoch"], _fmt(h["train_loss"], 6),
                               _fmt(h["val_loss"], 6), "yes" if h["epoch"] == best else ""])
        preds, _counts, _nwin = _predict(model, heads, tokenizer,
                                         [texts[i] for i in test_rows], tasks,
                                         windows=[windows[i] for i in test_rows],
                                         **predict_common)
        for name in tasks:
            for j, i in enumerate(test_rows):
                oof[name][i] = preds[name][j]
        for name, spec in tasks.items():
            actual = [(table[i].get(name) or "").strip() for i in test_rows]
            have = [j for j, v in enumerate(actual) if v]
            if spec["task"] == "regression":
                m = _regression_metrics([float(actual[j]) for j in have],
                                        [preds[name][j] for j in have])
                fold_rows.append([name, f + 1, m["n"], _fmt(m["r2"], rounding),
                                  _fmt(m["r"], rounding), _fmt(m["rho"], rounding),
                                  _fmt(m["mae"], rounding), "", "", ""])
            else:
                m = _classification_metrics([actual[j] for j in have],
                                            [preds[name][j] for j in have], spec["classes"])
                fold_rows.append([name, f + 1, m["n"], "", "", "", "",
                                  _fmt(m["accuracy"], rounding), _fmt(m["auc"], rounding),
                                  _fmt(m["macro"]["f1"], rounding)])
        del model, heads
        if device_name.startswith("cuda"):
            torch.cuda.empty_cache()

    # now we pool the out-of-fold predictions and get metrics per outcome
    results: Dict[str, dict] = {}
    for name, spec in tasks.items():
        actual = [(r.get(name) or "").strip() for r in table]
        have = [i for i, v in enumerate(actual) if v and oof[name][i] is not None]
        if spec["task"] == "regression":
            m = _regression_metrics([float(actual[i]) for i in have], [oof[name][i] for i in have])
            per_fold = [r for r in fold_rows if r[0] == name]
            r2s = [float(r[3]) for r in per_fold if r[3] != ""]
            m["r2_folds"] = float(np.mean(r2s)) if r2s else float("nan")
            m["r2_folds_se"] = float(np.std(r2s, ddof=1) / np.sqrt(len(r2s))) if len(r2s) > 1 else float("nan")
        else:
            m = _classification_metrics([actual[i] for i in have], [oof[name][i] for i in have],
                                        spec["classes"])
            per_fold = [r for r in fold_rows if r[0] == name]
            accs = [float(r[7]) for r in per_fold if r[7] != ""]
            aucs = [float(r[8]) for r in per_fold if r[8] != ""]
            m["accuracy_folds"] = float(np.mean(accs)) if accs else float("nan")
            m["accuracy_folds_se"] = float(np.std(accs, ddof=1) / np.sqrt(len(accs))) if len(accs) > 1 else float("nan")
            m["auc_folds"] = float(np.mean(aucs)) if aucs else float("nan")
            m["auc_folds_se"] = float(np.std(aucs, ddof=1) / np.sqrt(len(aucs))) if len(aucs) > 1 else float("nan")
        results[name] = m

    # the final model: every row, the median best epoch from the folds, and no
    # early stopping
    final_epochs = int(round(statistics.median(best_epochs))) if best_epochs else int(epochs)
    final_epochs = max(1, final_epochs)
    announce(on_progress, f"training the final model on all {len(table)} texts "
                          f"for {final_epochs} epoch(s)")
    final_common = dict(common)
    final_common.update(epochs=final_epochs, early_stopping=False)
    model, heads, final_history, _best = _train_one(
        resolved.source, tokenizer, table, tasks, weights, list(range(len(table))), [],
        seed=int(seed), warm_heads=warm, label="final model", **final_common)
    train_preds, counts, n_windows = _predict(model, heads, tokenizer, texts, tasks,
                                              windows=windows, **predict_common)

    # Save the model. a named model is a file of that name (the Train task
    # names one after its encoder: distilroberta-base-finetuned); an unnamed
    # one is named after what it predicts, as it always was
    stem = (slug(model_name, fallback="text_predictor") if model_name
            else f"text_predictor__{slug('_'.join(outcome_cols), fallback='outcomes')}")
    models_dir.mkdir(parents=True, exist_ok=True)
    manifest = models_dir / f"{stem}.json"
    payload = models_dir / f"{stem}.predictor"
    if payload.exists():
        import shutil

        shutil.rmtree(payload)
    model.eval()
    model.save_pretrained(payload / "encoder", safe_serialization=True)
    tokenizer.save_pretrained(payload / "encoder")
    from safetensors.torch import save_file

    head_tensors = {f"{_key(n)}.weight": heads[_key(n)][1].weight.detach().cpu().contiguous()
                    for n in tasks}
    head_tensors.update({f"{_key(n)}.bias": heads[_key(n)][1].bias.detach().cpu().contiguous()
                         for n in tasks})
    save_file(head_tensors, str(payload / "heads.safetensors"))
    wall = round(time.time() - started, 1)
    outcomes_doc = {}
    for name, spec in tasks.items():
        entry = {"task": spec["task"], "head": _key(name)}
        if spec["task"] == "regression":
            entry.update(mean=spec["mean"], std=spec["std"],
                         cv={k: (None if isinstance(v, float) and math.isnan(v) else v)
                             for k, v in results[name].items()
                             if k in ("r2", "r", "r_p", "rho", "rho_p", "rmse", "mae",
                                      "baseline_mae", "n", "r2_folds", "r2_folds_se")})
        else:
            entry.update(classes=spec["classes"], counts=spec["counts"],
                         cv={k: (None if isinstance(v, float) and math.isnan(v) else v)
                             for k, v in results[name].items()
                             if k in ("accuracy", "baseline_accuracy", "auc", "log_loss",
                                      "n", "accuracy_folds", "accuracy_folds_se",
                                      "auc_folds", "auc_folds_se")}
                         | {"f1_macro": results[name]["macro"]["f1"]})
        outcomes_doc[name] = entry
    doc = {
        "kind": PREDICTOR_KIND, "format": PREDICTOR_FORMAT,
        "name": model_name or stem,
        "payload": [payload.name],
        "payload_digests": payload_digests(payload),
        "base_model": resolved.base_model if resolved.kind != "hub" else resolved.source,
        "started_from": resolved.label,
        "reused_heads": reused_heads,
        "encoder": {"layers": layers, "pooling": pooling, "max_length": int(max_length),
                    "num_hidden_layers": n_layers, "hidden_size": hidden_size,
                    "parameters": n_params},
        "outcomes": outcomes_doc,
        "task_weights": weights,
        "training": {
            "n_texts": len(table), "n_folds": int(n_folds), "stratify": bool(stratify),
            "epochs": int(epochs), "best_epoch_per_fold": best_epochs,
            "final_epochs": final_epochs, "early_stopping": bool(early_stopping),
            "val_fraction": float(val_fraction), "learning_rate": float(learning_rate),
            "batch_size": int(batch_size), "grad_accum": int(grad_accum),
            "weight_decay": float(weight_decay), "warmup_fraction": float(warmup_fraction),
            "train_layers": int(train_layers),
            "gradient_checkpointing": bool(gradient_checkpointing),
            "seed": int(seed), "device": device_name, "precision": precision,
            "threads": threads,
            "windowed_share": sum(1 for k in n_windows if k > 1) / max(1, len(n_windows)),
            "final_history": final_history, "wall_seconds": wall,
        },
        "apply": {"batch_size": int(batch_size), "max_length": int(max_length),
                  "precision": precision, "emit_probabilities": True},
    }
    with atomic_write(manifest, mode="w", encoding="utf-8") as fh:
        json.dump(doc, fh, indent=1)

    # the tables
    announce(on_progress, "writing the results")
    metrics_header = ["feature_set", "n_feature_sets", "n_features", "outcome", "model",
                      "task", "n_used", "n_classes", "n_folds", "epochs_used", "base_model",
                      "cv_r2", "cv_r2_folds", "cv_r2_folds_se", "cv_r", "cv_r_p", "cv_rho",
                      "cv_rho_p", "cv_rmse", "cv_mae", "baseline_mae",
                      "accuracy", "baseline_accuracy", "accuracy_folds", "accuracy_folds_se",
                      "auc", "auc_folds", "auc_folds_se", "f1_macro", "f1_weighted",
                      "precision_macro", "recall_macro", "log_loss"]
    metric_rows = []
    for name, spec in tasks.items():
        m = results[name]
        row = ["text", 1, hidden_size, name, "fine-tuned transformer", spec["task"],
               m.get("n"), len(spec["classes"]) if spec["task"] == "classification" else "",
               int(n_folds), final_epochs, doc["base_model"]]
        if spec["task"] == "regression":
            row += [_fmt(m.get(k), rounding) for k in
                    ("r2", "r2_folds", "r2_folds_se", "r", "r_p", "rho", "rho_p", "rmse",
                     "mae", "baseline_mae")] + [""] * 12
        else:
            row += [""] * 10
            row += [_fmt(m.get("accuracy"), rounding), _fmt(m.get("baseline_accuracy"), rounding),
                    _fmt(m.get("accuracy_folds"), rounding), _fmt(m.get("accuracy_folds_se"), rounding),
                    _fmt(m.get("auc"), rounding), _fmt(m.get("auc_folds"), rounding),
                    _fmt(m.get("auc_folds_se"), rounding), _fmt(m["macro"]["f1"], rounding),
                    _fmt(m["weighted"]["f1"], rounding), _fmt(m["macro"]["precision"], rounding),
                    _fmt(m["macro"]["recall"], rounding), _fmt(m.get("log_loss"), rounding)]
        metric_rows.append(row)
    _write_csv(metrics_path, metrics_header, metric_rows, encoding)
    _write_csv(out_dir / "text_predictor_folds.csv",
               ["outcome", "fold", "n", "r2", "r", "rho", "mae", "accuracy", "auc", "f1_macro"],
               fold_rows, encoding)
    _write_csv(out_dir / "text_predictor_epochs.csv",
               ["fold", "epoch", "train_loss", "val_loss", "kept"], epoch_rows, encoding)
    class_rows, conf_rows = [], []
    for name, spec in tasks.items():
        if spec["task"] != "classification":
            continue
        m = results[name]
        for c in spec["classes"]:
            pc = m["per_class"][c]
            class_rows.append([name, class_label(doc, name, c), pc["support"],
                               _fmt(pc["precision"], rounding), _fmt(pc["recall"], rounding),
                               _fmt(pc["f1"], rounding), _fmt(pc.get("auc_vs_rest"), rounding)])
        for (t, p), n_ in m["confusion"].items():
            conf_rows.append([name, class_label(doc, name, t), class_label(doc, name, p), n_])
    if class_rows:
        _write_csv(out_dir / "text_predictor_per_class.csv",
                   ["outcome", "class", "support", "precision", "recall", "f1", "auc_vs_rest"],
                   class_rows, encoding)
        _write_csv(out_dir / "text_predictor_confusion.csv",
                   ["outcome", "true_class", "predicted_class", "n"], conf_rows, encoding)
    # the predictions: observed, out-of-fold, and the final model's own
    pred_header = ["text_id"]
    for name, spec in tasks.items():
        label = output_label(doc, name)
        pred_header += [name, f"oof_{label}", f"fold_{label}", f"pred_{label}"]
        if spec["task"] == "classification":
            pred_header += [f"prob_{label}"] + [f"p_{label}_{class_label(doc, name, c)}"
                                                 for c in spec["classes"]]
    pred_rows = []
    for i in range(len(table)):
        row: list = [ids[i]]
        for name, spec in tasks.items():
            observed = (table[i].get(name) or "").strip()
            o, p = oof[name][i], train_preds[name][i]
            if spec["task"] == "regression":
                row += [observed, _fmt(o, rounding), int(folds[i]) + 1, _fmt(p, rounding)]
            else:
                cls = spec["classes"]
                o_lab = class_label(doc, name, cls[int(np.argmax(o))]) if o is not None else ""
                p_lab = class_label(doc, name, cls[int(np.argmax(p))])
                row += [class_label(doc, name, observed) if observed else "", o_lab,
                        int(folds[i]) + 1, p_lab, _fmt(max(p), rounding)]
                row += [_fmt(v, rounding) for v in p]
        pred_rows.append(row)
    _write_csv(out_dir / f"text_predictor_predictions__{slug('_'.join(outcome_cols), fallback='outcomes')}.csv",
               pred_header, pred_rows, encoding)

    # lastly, the figures, the report section, and the report beside the model
    figures = _figures(out_dir, doc, tasks, results, table, oof, epoch_rows, final_history)
    from ..stats._common import write_section

    write_section(out_dir, SECTION_SLUG,
                  _section_md(doc, tasks, results, figures, out_dir, fallback))
    _write_report(manifest.with_name(f"{stem}_report.md"), doc, manifest, payload, tasks,
                  results, figures, out_dir, fallback)
    if verbose:
        for name, spec in tasks.items():
            m = results[name]
            head = (f"R² {m['r2']:.3f}, r {m['r']:.3f}" if spec["task"] == "regression"
                    else f"accuracy {m['accuracy']:.3f} (baseline {m['baseline_accuracy']:.3f}), "
                         f"AUC {m['auc']:.3f}")
            print(f"[text_predictor] {name}: out-of-fold {head}")
        print(f"[text_predictor] model -> {manifest}; results -> {out_dir}")
    return metrics_path

parse_task_weights

parse_task_weights(spec, outcomes)

"age: 1, condition: 2" -> weights per outcome, 1.0 where unsaid. An unknown outcome is refused by name.

Source code in src\taters\text\finetune_predictor.py

def parse_task_weights(spec: Union[str, Dict[str, float], None],
                       outcomes: Sequence[str]) -> Dict[str, float]:
    """``"age: 1, condition: 2"`` -> weights per outcome, 1.0 where unsaid.
    An unknown outcome is refused by name."""
    weights = {o: 1.0 for o in outcomes}
    if not spec:
        return weights
    items = spec.items() if isinstance(spec, dict) else (
        p.partition(":")[::2] for p in str(spec).split(",") if p.strip())
    for name, value in items:
        name = str(name).strip()
        if name not in weights:
            raise ValueError(f"task_weights names {name!r}, which is not one of the "
                             f"outcomes ({', '.join(outcomes)})")
        try:
            weights[name] = float(value)
        except (TypeError, ValueError):
            raise ValueError(f"the weight for {name!r} must be a number, not "
                             f"{value!r}") from None
        if weights[name] <= 0:
            raise ValueError(f"the weight for {name!r} must be positive")
    return weights

taters.text.adapt_encoder

Domain-adaptive pretraining: continue a language model's own training on your texts, so it speaks their dialect before it is asked to embed or predict anything.

A pre-trained encoder learned from web text and books. A corpus of bereavement-forum posts, clinical notes or adolescents' diaries uses words the encoder rarely saw and uses familiar words differently; adaptation continues the masked-language-model objective -- hide fifteen percent of the tokens, predict them -- on the corpus itself, with no labels, and leaves an encoder whose held-out perplexity on that corpus has fallen. That encoder is then the base for embeddings (:mod:transformer_embeddings) or for fine-tuning a predictor (:mod:finetune_predictor).

• Gururangan, S., et al. (2020). Don't stop pretraining: Adapt language models to domains and tasks. ACL 2020.

The loop is plain torch: AdamW with linear warm-up and decay, gradient clipping, half precision on a GPU, dynamic masking from transformers' collator, a held-out split by text whose loss and perplexity are measured before and after with the same masking seed, so the two numbers are comparable. A GPU that runs out of memory halves the batch and doubles the accumulation, keeping the effective batch the same.

What it writes: <name>.json (a taters-encoder manifest), the checkpoint folder <name>.encoder beside it, and <name>_report.md with everything a methods section needs.

adapt_encoder

adapt_encoder(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    out_model_json=None,
    out_report_md=None,
    overwrite_existing=False,
    workers=0,
    on_progress=None,
    verbose=True,
    encoding="utf-8-sig",
    delimiter=",",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    base_model=CURATED_ENCODERS[0][0],
    name=None,
    epochs=3,
    max_length=256,
    batch_size=16,
    grad_accum=2,
    learning_rate=5e-05,
    warmup_fraction=0.06,
    weight_decay=0.01,
    mlm_probability=0.15,
    heldout_fraction=0.1,
    train_layers=0,
    gradient_checkpointing=False,
    device="auto",
    precision="auto",
    seed=42
)

Continue an encoder's masked-language-model training on these texts.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The same input contract as every other text step: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
txt_dir	Optional[PathLike]	The same input contract as every other text step: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
analysis_csv	Optional[PathLike]	The same input contract as every other text step: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
gathered_csv	Optional[PathLike]	The same input contract as every other text step: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
out_model_json	str or Path	The encoder manifest; the checkpoint lands beside it as <stem>.encoder. Default ./models/adapted_encoder.json.	None
out_report_md	str or Path	The training report, default <stem>_report.md beside the model.	None
overwrite_existing	bool	If False and the manifest exists, return it untouched.	False
workers	int	Parallel processes for the gather, and the CPU threads torch may use. 0 means automatic.	0
text_cols	sequence of str	When gathering from a CSV, the column(s) holding the text.	("text",)
id_cols	sequence of str	Columns that identify each row when gathering from a CSV.	None
mode	('concat', 'separate')	With several text columns: join them into one text per row, or treat each as its own text.	"concat"
group_by	sequence of str	Columns to combine rows by before training (one text per group).	None
pattern	str	Which files to read when gathering from a folder of documents.	every document type
base_model	str	The encoder to start from: a Hugging Face name, a checkpoint folder, or a Taters text encoder file (adapting twice is allowed).	CURATED_ENCODERS[0][0]
name	str	The encoder's name in menus; default the manifest's file stem.	None
epochs	int	Passes over the corpus. One to three is usual for adaptation; the report's held-out loss per epoch shows when more stopped helping.	3
max_length	int	Tokens per training window. Longer texts are cut into windows so every token is trained on; 256 is a good trade of context for speed.	256
batch_size	int	Windows per forward pass; halved automatically if the GPU runs out of memory, with grad_accum doubled to compensate.	16
grad_accum	int	Batches accumulated per optimizer step. The effective batch is batch_size × grad_accum.	2
learning_rate	float	The peak learning rate of AdamW, after warm-up.	5e-5
warmup_fraction	float	The share of optimizer steps spent warming the learning rate up linearly from zero; it then decays linearly to zero.	0.06
weight_decay	float	AdamW's weight decay.	0.01
mlm_probability	float	The share of tokens masked in each window, freshly drawn each pass.	0.15
heldout_fraction	float	The share of texts set aside, never trained on, to measure the loss and perplexity before and after.	0.1
train_layers	int	Train only the top this-many layers (and the prediction head); 0 trains everything. Two layers is a good CPU compromise.	0
gradient_checkpointing	bool	Trade compute for memory on a small card.	False
device	('auto', 'cuda', 'cpu')	Where training runs.	"auto"
precision	('auto', 'fp32', 'fp16')	Half precision on a GPU (auto), always full, or always half.	"auto"
seed	int	Seeds the held-out split, the shuffles, the masking and the initialization of anything new.	42

Returns:

Type	Description
Path	out_model_json.

Source code in src\taters\text\adapt_encoder.py

def adapt_encoder(
    *,
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    out_model_json: Optional[PathLike] = None,
    out_report_md: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    workers: int = 0,
    on_progress: Optional[Callable[..., None]] = None,
    verbose: bool = True,
    encoding: str = "utf-8-sig",
    delimiter: str = ",",
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ----- what to adapt, and how -----
    base_model: str = CURATED_ENCODERS[0][0],
    name: Optional[str] = None,
    epochs: int = 3,
    max_length: int = 256,
    batch_size: int = 16,
    grad_accum: int = 2,
    learning_rate: float = 5e-5,
    warmup_fraction: float = 0.06,
    weight_decay: float = 0.01,
    mlm_probability: float = 0.15,
    heldout_fraction: float = 0.1,
    train_layers: int = 0,
    gradient_checkpointing: bool = False,
    device: Literal["auto", "cuda", "cpu"] = "auto",
    precision: Literal["auto", "fp32", "fp16"] = "auto",
    seed: int = 42,
) -> Path:
    """
    Continue an encoder's masked-language-model training on these texts.

    Parameters
    ----------
    csv_path, txt_dir, analysis_csv, gathered_csv
        The same input contract as every other text step: a spreadsheet of
        texts, a folder of documents, or a prebuilt analysis-ready CSV.
    out_model_json : str or Path, optional
        The encoder manifest; the checkpoint lands beside it as
        ``<stem>.encoder``. Default ``./models/adapted_encoder.json``.
    out_report_md : str or Path, optional
        The training report, default ``<stem>_report.md`` beside the model.
    overwrite_existing : bool, default False
        If False and the manifest exists, return it untouched.
    workers : int, default 0
        Parallel processes for the gather, and the CPU threads torch may
        use. 0 means automatic.
    text_cols : sequence of str, default ("text",)
        When gathering from a CSV, the column(s) holding the text.
    id_cols : sequence of str, optional
        Columns that identify each row when gathering from a CSV.
    mode : {"concat", "separate"}, default "concat"
        With several text columns: join them into one text per row, or
        treat each as its own text.
    group_by : sequence of str, optional
        Columns to combine rows by before training (one text per group).
    pattern : str, default every document type
        Which files to read when gathering from a folder of documents.
    base_model : str
        The encoder to start from: a Hugging Face name, a checkpoint
        folder, or a Taters text encoder file (adapting twice is allowed).
    name : str, optional
        The encoder's name in menus; default the manifest's file stem.
    epochs : int, default 3
        Passes over the corpus. One to three is usual for adaptation; the
        report's held-out loss per epoch shows when more stopped helping.
    max_length : int, default 256
        Tokens per training window. Longer texts are cut into windows so
        every token is trained on; 256 is a good trade of context for speed.
    batch_size : int, default 16
        Windows per forward pass; halved automatically if the GPU runs out
        of memory, with ``grad_accum`` doubled to compensate.
    grad_accum : int, default 2
        Batches accumulated per optimizer step. The effective batch is
        ``batch_size × grad_accum``.
    learning_rate : float, default 5e-5
        The peak learning rate of AdamW, after warm-up.
    warmup_fraction : float, default 0.06
        The share of optimizer steps spent warming the learning rate up
        linearly from zero; it then decays linearly to zero.
    weight_decay : float, default 0.01
        AdamW's weight decay.
    mlm_probability : float, default 0.15
        The share of tokens masked in each window, freshly drawn each pass.
    heldout_fraction : float, default 0.1
        The share of *texts* set aside, never trained on, to measure the
        loss and perplexity before and after.
    train_layers : int, default 0
        Train only the top this-many layers (and the prediction head); 0
        trains everything. Two layers is a good CPU compromise.
    gradient_checkpointing : bool, default False
        Trade compute for memory on a small card.
    device : {"auto", "cuda", "cpu"}, default "auto"
        Where training runs.
    precision : {"auto", "fp32", "fp16"}, default "auto"
        Half precision on a GPU (auto), always full, or always half.
    seed : int, default 42
        Seeds the held-out split, the shuffles, the masking and the
        initialization of anything new.

    Returns
    -------
    Path
        ``out_model_json``.
    """
    missing = torch_missing_reason()
    if missing:
        raise ImportError(missing)
    if int(epochs) < 1 or int(batch_size) < 1 or int(grad_accum) < 1:
        raise ValueError("epochs, batch_size and grad_accum must be at least 1")
    if not (0.0 < float(mlm_probability) < 1.0):
        raise ValueError("mlm_probability must be between 0 and 1")
    if not (0.0 <= float(heldout_fraction) < 1.0):
        raise ValueError("heldout_fraction must be at least 0 and below 1")
    started = time.time()

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)
    out_model_json = Path(out_model_json) if out_model_json else \
        Path.cwd() / "models" / "adapted_encoder.json"
    if out_model_json.is_file() and not overwrite_existing:
        if verbose:
            print("Adapted encoder already exists; returning existing file.")
        return out_model_json
    out_report_md = Path(out_report_md) if out_report_md else \
        out_model_json.with_name(out_model_json.stem + "_report.md")
    out_model_json.parent.mkdir(parents=True, exist_ok=True)

    import csv

    from ..helpers.csvio import widen_csv_field_limit

    widen_csv_field_limit()
    ids: List[str] = []
    texts: List[str] = []
    with analysis_ready.open("r", newline="", encoding=encoding) as fh:
        for row in csv.DictReader(fh, delimiter=delimiter):
            text = (row.get("text") or "").strip()
            if text:
                ids.append(str(row.get("text_id", "")))
                texts.append(text)
    if len(texts) < 2:
        raise ValueError(
            f"only {len(texts)} text(s) with words in them; adapting an encoder "
            f"needs a corpus (at least two texts, ideally thousands).")

    import torch

    torch.manual_seed(int(seed))
    threads = set_threads(workers)
    announce(on_progress, f"loading {base_model}")
    model, tokenizer, resolved, device_name, reason = load_encoder(
        base_model, device=device, for_mlm=True, verbose=verbose)
    if gradient_checkpointing and hasattr(model, "gradient_checkpointing_enable"):
        model.gradient_checkpointing_enable()
    trainable, frozen = freeze_below(model, int(train_layers))

    run = train_mlm(
        model=model, tokenizer=tokenizer, texts=texts, ids=ids,
        device_name=device_name, precision=precision, seed=int(seed),
        epochs=int(epochs), max_length=int(max_length),
        batch_size=int(batch_size), grad_accum=int(grad_accum),
        learning_rate=float(learning_rate), warmup_fraction=float(warmup_fraction),
        weight_decay=float(weight_decay), mlm_probability=float(mlm_probability),
        heldout_fraction=float(heldout_fraction),
        # adaptation is the fall from before to after, so before is measured;
        # and it runs the epochs it was asked for, the report showing when
        # more stopped helping
        measure_before=True, early_stopping=False,
        verbose=verbose, on_progress=on_progress,
        announce=lambda phrase: announce(on_progress, phrase))
    loss_before, loss_after = run.loss_before, run.loss_after
    train_idx, held_idx, token_counts = run.train_idx, run.held_idx, run.token_counts

    # lastly, we save: the checkpoint folder goes beside the manifest, and
    # then the manifest itself
    announce(on_progress, "saving the adapted encoder")
    folder = out_model_json.with_suffix(".encoder")
    if folder.exists():
        import shutil

        shutil.rmtree(folder)
    model.eval()
    model.save_pretrained(folder, safe_serialization=True)
    tokenizer.save_pretrained(folder)
    cfg = model.config
    model_name = str(name or out_model_json.stem)
    wall = round(time.time() - started, 1)
    doc = {
        "kind": ENCODER_KIND, "format": ENCODER_FORMAT, "name": model_name,
        "payload": [folder.name],
        "payload_digests": payload_digests(folder),
        "base_model": resolved.base_model if resolved.kind != "hub" else resolved.source,
        "adapted_from": resolved.label,
        "architecture": str(getattr(cfg, "model_type", "")),
        "num_hidden_layers": int(getattr(cfg, "num_hidden_layers", 0) or getattr(cfg, "n_layers", 0)),
        "hidden_size": int(getattr(cfg, "hidden_size", 0) or getattr(cfg, "dim", 0)),
        "vocab_size": int(getattr(cfg, "vocab_size", 0)),
        "parameters": int(sum(p.numel() for p in model.parameters())),
        "text": {"max_length": int(max_length)},
        "training": {
            "epochs": int(epochs), "max_length": int(max_length),
            "batch_size": int(batch_size), "grad_accum": int(grad_accum),
            "final_batch_size": run.final_batch_size, "final_grad_accum": run.final_grad_accum,
            "oom_restarts": run.oom_restarts,
            "learning_rate": float(learning_rate),
            "warmup_fraction": float(warmup_fraction),
            "weight_decay": float(weight_decay),
            "mlm_probability": float(mlm_probability),
            "heldout_fraction": float(heldout_fraction),
            "train_layers": int(train_layers), "trainable_parameters": trainable,
            "frozen_parameters": frozen,
            "gradient_checkpointing": bool(gradient_checkpointing),
            "seed": int(seed), "device": device_name, "precision": precision,
            "threads": threads, "optimizer_steps": run.optimizer_steps,
            "n_texts": len(texts), "n_train_texts": len(train_idx),
            "n_heldout_texts": len(held_idx), "n_tokens": int(sum(token_counts)),
            "n_train_windows": run.n_train_windows, "n_heldout_windows": run.n_heldout_windows,
            "windowed_share": run.windowed_share,
            "step_loss": [round(x, 5) for x in run.step_losses],
            "learning_rate_trace": [float(f"{x:.3g}") for x in run.lr_trace],
            "heldout_loss_per_epoch": [round(x, 5) for x in run.heldout_per_epoch],
            "wall_seconds": wall,
        },
        "evaluation": {
            "loss_before": round(loss_before, 5), "loss_after": round(loss_after, 5),
            "perplexity_before": round(math.exp(loss_before), 3),
            "perplexity_after": round(math.exp(loss_after), 3),
            "heldout_texts": len(held_idx),
        },
        # how this encoder gets read when it embeds text; people can edit this
        # per model
        "apply": {"layers": "second_to_last", "pooling": "mean"},
    }
    with atomic_write(out_model_json, mode="w", encoding="utf-8") as fh:
        json.dump(doc, fh, indent=1)
    _write_report(out_report_md, doc, out_model_json, folder,
                  mismatch=_tokenizer_mismatch(texts, tokenizer),
                  device_name=device_name, fallback=reason)
    if verbose:
        ev = doc["evaluation"]
        print(f"[adapt_encoder] {model_name}: held-out perplexity "
              f"{ev['perplexity_before']:.2f} -> {ev['perplexity_after']:.2f} "
              f"after {epochs} epoch(s) on {len(train_idx)} text(s) -> {out_model_json}")
    return out_model_json

split_heldout

split_heldout(text_ids, fraction, seed)

Indices of the training and held-out texts: a seeded shuffle, split by text, never by chunk, so no held-out sentence has a neighbor from the same document in the training set inflating the after-score. At least one text is held out whenever there are two or more.

Source code in src\taters\text\_mlm_train.py

def split_heldout(text_ids: Sequence[str], fraction: float, seed: int
                  ) -> Tuple[List[int], List[int]]:
    """
    Indices of the training and held-out texts: a seeded shuffle, split by
    *text*, never by chunk, so no held-out sentence has a neighbor from the
    same document in the training set inflating the after-score. At least
    one text is held out whenever there are two or more.
    """
    n = len(text_ids)
    order = list(range(n))
    random.Random(int(seed)).shuffle(order)
    k = int(round(n * float(fraction)))
    if n >= 2:
        k = min(max(1, k), n - 1)
    else:
        k = 0
    return sorted(order[k:]), sorted(order[:k])

taters.text.pretrain_encoder

Pretrain a transformer encoder from scratch on your own texts.

Adapting (:mod:adapt_encoder) starts from a model somebody else pretrained and continues its training on your corpus. This starts from nothing: random weights, and a tokenizer built from your texts rather than borrowed from web text -- so a corpus of clinical notes, forum posts or eighteenth-century letters gets a vocabulary of its own words, whole, instead of one that shatters them into pieces.

It is heavy-duty, and the step says so before it runs. A language model learns language from quantity: below a few million words of text the result will be worse at everything than any pretrained model you could have adapted, and even a modest model takes hours on one GPU and days on a CPU. The corpus size is left to your judgment -- testing the machinery on a small one is a legitimate thing to do -- but the report says what it was trained on, so nobody mistakes a toy for a tool.

The objective is masked-language modeling (Devlin et al., 2019) in the RoBERTa style (Liu et al., 2019): byte-level BPE, dynamic masking, no next-sentence task. The loop is the one adaptation uses (:mod:_mlm_train), with two things pretraining needs and adaptation does not: early stopping on the held-out loss, keeping the best epoch, and a learning-rate schedule sized for a model that knows nothing yet.

• Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL 2019.
• Liu, Y., et al. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692.

What it writes: <name>.json (a taters-encoder manifest, the same kind adaptation writes, so the library, the embeddings step and fine-tuning treat it like any other encoder), the checkpoint folder <name>.encoder beside it, and <name>_report.md.

pretrain_encoder

pretrain_encoder(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    out_model_json=None,
    out_report_md=None,
    overwrite_existing=False,
    workers=0,
    on_progress=None,
    verbose=True,
    encoding="utf-8-sig",
    delimiter=",",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    name=None,
    preset="small",
    vocab_size=None,
    layers=None,
    hidden_size=None,
    attention_heads=None,
    max_length=256,
    epochs=40,
    patience=3,
    batch_size=32,
    grad_accum=2,
    learning_rate=0.0005,
    warmup_fraction=0.1,
    weight_decay=0.01,
    mlm_probability=0.15,
    heldout_fraction=0.1,
    gradient_checkpointing=False,
    device="auto",
    precision="auto",
    seed=42
)

Train a transformer encoder, and its tokenizer, from nothing but these texts.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The same input contract as every other text step: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
txt_dir	Optional[PathLike]	The same input contract as every other text step: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
analysis_csv	Optional[PathLike]	The same input contract as every other text step: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
gathered_csv	Optional[PathLike]	The same input contract as every other text step: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
out_model_json	str or Path	The encoder manifest; the checkpoint lands beside it as <stem>.encoder. Default ./models/scratch_encoder.json.	None
out_report_md	str or Path	The training report, default <stem>_report.md beside the model.	None
overwrite_existing	bool	If False and the manifest exists, return it untouched.	False
workers	int	Parallel processes for the gather, and the CPU threads torch may use. 0 means automatic.	0
text_cols	Sequence[str]	Gather options, as in every text step.	('text',)
id_cols	Sequence[str]	Gather options, as in every text step.	('text',)
mode	Sequence[str]	Gather options, as in every text step.	('text',)
group_by	Sequence[str]	Gather options, as in every text step.	('text',)
pattern	Sequence[str]	Gather options, as in every text step.	('text',)
name	str	The encoder's name in menus; default the manifest's file stem.	None
preset	('small', 'base', 'custom')	The architecture. small is 4 layers, 256 wide (about 10M parameters, an overnight run on one GPU); base is the BERT/RoBERTa shape, 12 layers, 768 wide (about 110M, days). custom takes layers, hidden_size and attention_heads.	"small"
vocab_size	int	Symbols in the tokenizer. Default follows the preset (8,000 for small, 30,000 for base). At least 300.	None
layers	int	The numbers behind a custom preset; ignored otherwise. The feed-forward width is four times hidden_size, as it is in BERT.	None
hidden_size	int	The numbers behind a custom preset; ignored otherwise. The feed-forward width is four times hidden_size, as it is in BERT.	None
attention_heads	int	The numbers behind a custom preset; ignored otherwise. The feed-forward width is four times hidden_size, as it is in BERT.	None
max_length	int	Tokens per training window, and the longest input the finished model will read. Longer texts are cut into windows for training.	256
epochs	int	The most passes over the corpus. Training stops earlier when the held-out loss has not fallen for patience epochs, and keeps the weights from the best epoch.	40
patience	int	How many epochs without improvement end the run.	3
batch_size	int	Windows per forward pass; halved automatically if the GPU runs out of memory, with grad_accum scaled to compensate.	32
grad_accum	int	Batches accumulated per optimizer step.	2
learning_rate	float	The peak learning rate of AdamW -- ten times adaptation's, because there is nothing here worth preserving yet.	5e-4
warmup_fraction	float	The share of optimizer steps warming the learning rate up from zero.	0.1
weight_decay	float	AdamW's weight decay.	0.01
mlm_probability	float	The share of tokens masked in each window, freshly drawn each pass.	0.15
heldout_fraction	float	The share of texts never trained on -- and never shown to the tokenizer -- whose loss decides when to stop.	0.1
gradient_checkpointing	bool	Trade compute for memory on a small card.	False
device	('auto', 'cuda', 'cpu')	Where training runs. With more than one GPU visible, all of them are used from this one process (torch.nn.DataParallel).	"auto"
precision	('auto', 'fp32', 'fp16')	Half precision on a GPU (auto), always full, or always half.	"auto"
seed	int	Seeds the weights, the held-out split, the shuffles and the masks.	42

Returns:

Type	Description
Path	out_model_json.

Source code in src\taters\text\pretrain_encoder.py

def pretrain_encoder(
    *,
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    out_model_json: Optional[PathLike] = None,
    out_report_md: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    workers: int = 0,
    on_progress: Optional[Callable[..., None]] = None,
    verbose: bool = True,
    encoding: str = "utf-8-sig",
    delimiter: str = ",",
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ----- what to build -----
    name: Optional[str] = None,
    preset: str = "small",
    vocab_size: Optional[int] = None,
    layers: Optional[int] = None,
    hidden_size: Optional[int] = None,
    attention_heads: Optional[int] = None,
    max_length: int = 256,

    # ----- how to train it -----
    epochs: int = 40,
    patience: int = 3,
    batch_size: int = 32,
    grad_accum: int = 2,
    learning_rate: float = 5e-4,
    warmup_fraction: float = 0.1,
    weight_decay: float = 0.01,
    mlm_probability: float = 0.15,
    heldout_fraction: float = 0.1,
    gradient_checkpointing: bool = False,
    device: Literal["auto", "cuda", "cpu"] = "auto",
    precision: Literal["auto", "fp32", "fp16"] = "auto",
    seed: int = 42,
) -> Path:
    """
    Train a transformer encoder, and its tokenizer, from nothing but these texts.

    Parameters
    ----------
    csv_path, txt_dir, analysis_csv, gathered_csv
        The same input contract as every other text step: a spreadsheet of
        texts, a folder of documents, or a prebuilt analysis-ready CSV.
    out_model_json : str or Path, optional
        The encoder manifest; the checkpoint lands beside it as
        ``<stem>.encoder``. Default ``./models/scratch_encoder.json``.
    out_report_md : str or Path, optional
        The training report, default ``<stem>_report.md`` beside the model.
    overwrite_existing : bool, default False
        If False and the manifest exists, return it untouched.
    workers : int, default 0
        Parallel processes for the gather, and the CPU threads torch may
        use. 0 means automatic.
    text_cols, id_cols, mode, group_by, pattern
        Gather options, as in every text step.
    name : str, optional
        The encoder's name in menus; default the manifest's file stem.
    preset : {"small", "base", "custom"}, default "small"
        The architecture. ``small`` is 4 layers, 256 wide (about 10M
        parameters, an overnight run on one GPU); ``base`` is the
        BERT/RoBERTa shape, 12 layers, 768 wide (about 110M, days).
        ``custom`` takes ``layers``, ``hidden_size`` and ``attention_heads``.
    vocab_size : int, optional
        Symbols in the tokenizer. Default follows the preset (8,000 for
        small, 30,000 for base). At least 300.
    layers, hidden_size, attention_heads : int, optional
        The numbers behind a ``custom`` preset; ignored otherwise. The
        feed-forward width is four times ``hidden_size``, as it is in BERT.
    max_length : int, default 256
        Tokens per training window, and the longest input the finished
        model will read. Longer texts are cut into windows for training.
    epochs : int, default 40
        The *most* passes over the corpus. Training stops earlier when the
        held-out loss has not fallen for ``patience`` epochs, and keeps the
        weights from the best epoch.
    patience : int, default 3
        How many epochs without improvement end the run.
    batch_size : int, default 32
        Windows per forward pass; halved automatically if the GPU runs out
        of memory, with ``grad_accum`` scaled to compensate.
    grad_accum : int, default 2
        Batches accumulated per optimizer step.
    learning_rate : float, default 5e-4
        The peak learning rate of AdamW -- ten times adaptation's, because
        there is nothing here worth preserving yet.
    warmup_fraction : float, default 0.1
        The share of optimizer steps warming the learning rate up from zero.
    weight_decay : float, default 0.01
        AdamW's weight decay.
    mlm_probability : float, default 0.15
        The share of tokens masked in each window, freshly drawn each pass.
    heldout_fraction : float, default 0.1
        The share of *texts* never trained on -- and never shown to the
        tokenizer -- whose loss decides when to stop.
    gradient_checkpointing : bool, default False
        Trade compute for memory on a small card.
    device : {"auto", "cuda", "cpu"}, default "auto"
        Where training runs. With more than one GPU visible, all of them
        are used from this one process (``torch.nn.DataParallel``).
    precision : {"auto", "fp32", "fp16"}, default "auto"
        Half precision on a GPU (auto), always full, or always half.
    seed : int, default 42
        Seeds the weights, the held-out split, the shuffles and the masks.

    Returns
    -------
    Path
        ``out_model_json``.
    """
    missing = torch_missing_reason()
    if missing:
        raise ImportError(missing)
    if preset not in PRESETS:
        raise ValueError(f"preset must be one of {sorted(PRESETS)}, not {preset!r}")
    if int(epochs) < 1 or int(batch_size) < 1 or int(grad_accum) < 1 or int(patience) < 1:
        raise ValueError("epochs, patience, batch_size and grad_accum must be at least 1")
    if not (0.0 < float(mlm_probability) < 1.0):
        raise ValueError("mlm_probability must be between 0 and 1")
    if not (0.0 <= float(heldout_fraction) < 1.0):
        raise ValueError("heldout_fraction must be at least 0 and below 1")
    if int(max_length) < 16:
        raise ValueError("max_length must be at least 16 tokens")
    shape = dict(PRESETS[preset])
    if preset == "custom":
        for key, value in (("layers", layers), ("hidden_size", hidden_size),
                           ("attention_heads", attention_heads)):
            if value is None:
                raise ValueError(f"preset 'custom' needs {key}")
            shape[key] = int(value)
        if shape["hidden_size"] % shape["attention_heads"]:
            raise ValueError("hidden_size must be a multiple of attention_heads")
    vocab = int(vocab_size) if vocab_size is not None else int(shape["vocab_size"])
    if vocab < MIN_VOCAB:
        raise ValueError(f"vocab_size must be at least {MIN_VOCAB}")
    started = time.time()

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)
    out_model_json = Path(out_model_json) if out_model_json else \
        Path.cwd() / "models" / "scratch_encoder.json"
    if out_model_json.is_file() and not overwrite_existing:
        if verbose:
            print("Encoder already exists; returning existing file.")
        return out_model_json
    out_report_md = Path(out_report_md) if out_report_md else \
        out_model_json.with_name(out_model_json.stem + "_report.md")
    out_model_json.parent.mkdir(parents=True, exist_ok=True)

    import csv

    from ..helpers.csvio import widen_csv_field_limit

    widen_csv_field_limit()
    ids: List[str] = []
    texts: List[str] = []
    with analysis_ready.open("r", newline="", encoding=encoding) as fh:
        for row in csv.DictReader(fh, delimiter=delimiter):
            text = (row.get("text") or "").strip()
            if text:
                ids.append(str(row.get("text_id", "")))
                texts.append(text)
    if len(texts) < 2:
        raise ValueError(
            f"only {len(texts)} text(s) with words in them; pretraining an encoder "
            f"needs a corpus (at least two texts -- realistically, millions of words).")

    import torch

    torch.manual_seed(int(seed))
    threads = set_threads(workers)

    # the tokenizer learns from the training texts only. the split here is the
    # same seeded split the loop makes again below, so the two agree exactly.
    train_idx, held_idx = split_heldout(ids, heldout_fraction, seed)
    announce(on_progress, f"learning a {vocab:,}-symbol vocabulary from "
                          f"{len(train_idx)} texts")
    tokenizer = train_tokenizer([texts[i] for i in train_idx], vocab_size=vocab)
    # the finished model's real limit, so that anything reading it later
    # (the embeddings step, fine-tuning) windows at the right length
    tokenizer.model_max_length = int(max_length)

    announce(on_progress, f"building a {preset} encoder from random weights")
    config = preset_config(shape, vocab_size=tokenizer.vocab_size,
                           max_length=int(max_length), tokenizer=tokenizer)
    model, device_name, reason, devices = fresh_encoder(
        config, device=device, seed=int(seed), verbose=verbose)
    if gradient_checkpointing and hasattr(unwrap(model), "gradient_checkpointing_enable"):
        unwrap(model).gradient_checkpointing_enable()

    run = train_mlm(
        model=model, tokenizer=tokenizer, texts=texts, ids=ids,
        device_name=device_name, precision=precision, seed=int(seed),
        epochs=int(epochs), max_length=int(max_length),
        batch_size=int(batch_size), grad_accum=int(grad_accum),
        learning_rate=float(learning_rate), warmup_fraction=float(warmup_fraction),
        weight_decay=float(weight_decay), mlm_probability=float(mlm_probability),
        heldout_fraction=float(heldout_fraction),
        # a random model's "before" is a guess at the vocabulary size, not a
        # measurement worth reporting; and epochs is a ceiling, not a plan
        measure_before=False, early_stopping=True, patience=int(patience),
        verbose=verbose, on_progress=on_progress,
        announce=lambda phrase: announce(on_progress, phrase))
    assert run.train_idx == train_idx and run.held_idx == held_idx, \
        "the tokenizer and the loop split the corpus differently"

    announce(on_progress, "saving the encoder")
    folder = out_model_json.with_suffix(".encoder")
    if folder.exists():
        import shutil

        shutil.rmtree(folder)
    bare = unwrap(model)
    bare.eval()
    bare.save_pretrained(folder, safe_serialization=True)
    tokenizer.save_pretrained(folder)
    cfg = bare.config
    model_name = str(name or out_model_json.stem)
    wall = round(time.time() - started, 1)
    loss_final = run.loss_after
    doc = {
        "kind": ENCODER_KIND, "format": ENCODER_FORMAT, "name": model_name,
        "payload": [folder.name],
        "payload_digests": payload_digests(folder),
        # no base model: this is what tells a reader, and the library row,
        # that nothing here was pretrained by anyone else
        "trained_from": "scratch",
        "preset": preset,
        "architecture": str(getattr(cfg, "model_type", "")),
        "num_hidden_layers": int(cfg.num_hidden_layers),
        "hidden_size": int(cfg.hidden_size),
        "attention_heads": int(cfg.num_attention_heads),
        "vocab_size": int(cfg.vocab_size),
        "parameters": int(sum(p.numel() for p in bare.parameters())),
        "tokenizer": {"kind": "byte_bpe", "vocab_size": int(tokenizer.vocab_size),
                      "trained_on": "training texts", "min_frequency": 2},
        "text": {"max_length": int(max_length)},
        "training": {
            "epochs": int(epochs), "epochs_run": run.epochs_run,
            "best_epoch": run.best_epoch, "stopped_early": run.stopped_early,
            "patience": int(patience),
            "max_length": int(max_length),
            "batch_size": int(batch_size), "grad_accum": int(grad_accum),
            "final_batch_size": run.final_batch_size, "final_grad_accum": run.final_grad_accum,
            "oom_restarts": run.oom_restarts,
            "learning_rate": float(learning_rate),
            "warmup_fraction": float(warmup_fraction),
            "weight_decay": float(weight_decay),
            "mlm_probability": float(mlm_probability),
            "heldout_fraction": float(heldout_fraction),
            "gradient_checkpointing": bool(gradient_checkpointing),
            "seed": int(seed), "device": device_name, "devices": devices,
            "precision": precision, "threads": threads,
            "optimizer_steps": run.optimizer_steps,
            "n_texts": len(texts), "n_train_texts": len(run.train_idx),
            "n_heldout_texts": len(run.held_idx),
            "n_tokens": int(sum(run.token_counts)),
            "n_train_windows": run.n_train_windows, "n_heldout_windows": run.n_heldout_windows,
            "windowed_share": run.windowed_share,
            "step_loss": [round(x, 5) for x in run.step_losses],
            "learning_rate_trace": [float(f"{x:.3g}") for x in run.lr_trace],
            "heldout_loss_per_epoch": [round(x, 5) for x in run.heldout_per_epoch],
            "wall_seconds": wall,
        },
        "evaluation": {
            "loss_final": round(loss_final, 5),
            "perplexity_final": round(math.exp(loss_final), 3),
            "heldout_texts": len(run.held_idx),
        },
        # how this encoder gets read when it embeds text; people can edit this
        # per model
        "apply": {"layers": "second_to_last", "pooling": "mean"},
    }
    with atomic_write(out_model_json, mode="w", encoding="utf-8") as fh:
        json.dump(doc, fh, indent=1)
    _write_report(out_report_md, doc, out_model_json, folder,
                  device_name=device_name, fallback=reason)
    if verbose:
        ev = doc["evaluation"]
        print(f"[pretrain_encoder] {model_name}: held-out perplexity "
              f"{ev['perplexity_final']:.2f} after {run.epochs_run} epoch(s) "
              f"(best: {run.best_epoch}) on {len(run.train_idx)} text(s) -> {out_model_json}")
    return out_model_json

train_tokenizer

train_tokenizer(texts, *, vocab_size, min_frequency=2)

A byte-level BPE tokenizer learned from texts alone.

Byte-level, so no text can contain a character it cannot represent -- every byte is a symbol before any merge is learned -- and RoBERTa-style, so the model's config and the collator's masking agree with it without special handling.

Parameters:

Name	Type	Description	Default
texts	sequence of str	The texts to learn merges from. Pass the training texts only: a tokenizer that has seen the held-out texts has leaked a little of them into every measurement made on them.	required
vocab_size	int	How many symbols, byte symbols and special tokens included.	required
min_frequency	int	A merge has to occur this often to be kept.	2

Returns:

Type	Description
RobertaTokenizerFast

Source code in src\taters\text\pretrain_encoder.py

def train_tokenizer(texts: Sequence[str], *, vocab_size: int, min_frequency: int = 2):
    """
    A byte-level BPE tokenizer learned from ``texts`` alone.

    Byte-level, so no text can contain a character it cannot represent --
    every byte is a symbol before any merge is learned -- and RoBERTa-style,
    so the model's config and the collator's masking agree with it without
    special handling.

    Parameters
    ----------
    texts : sequence of str
        The texts to learn merges from. Pass the *training* texts only: a
        tokenizer that has seen the held-out texts has leaked a little of
        them into every measurement made on them.
    vocab_size : int
        How many symbols, byte symbols and special tokens included.
    min_frequency : int, default 2
        A merge has to occur this often to be kept.

    Returns
    -------
    transformers.RobertaTokenizerFast
    """
    from tokenizers import ByteLevelBPETokenizer
    from transformers import RobertaTokenizerFast

    if int(vocab_size) < MIN_VOCAB:
        raise ValueError(f"vocab_size must be at least {MIN_VOCAB}: a byte-level "
                         f"tokenizer starts with 256 byte symbols, and below "
                         f"{MIN_VOCAB} it has learned almost nothing beyond them.")
    bpe = ByteLevelBPETokenizer()
    bpe.train_from_iterator(list(texts), vocab_size=int(vocab_size),
                            min_frequency=int(min_frequency),
                            special_tokens=list(SPECIAL_TOKENS))
    bos, pad, eos, unk, mask = SPECIAL_TOKENS
    return RobertaTokenizerFast(tokenizer_object=bpe, bos_token=bos, eos_token=eos,
                                sep_token=eos, cls_token=bos, unk_token=unk,
                                pad_token=pad, mask_token=mask)

taters.text.transformer_embeddings

Embeddings from any transformer encoder, sentence by sentence, averaged per text.

Conceptually the sentence-transformers step: each text is split into sentences with the same splitter, every sentence is encoded, and the text's vector is the mean of its sentence vectors. What this step adds is the choice of encoder and of reading: any Hugging Face encoder, an encoder adapted to your corpus in Taters, or the encoder inside a fine-tuned predictor; which hidden layers are read and how token vectors are pooled. A sentence-transformers model has been trained to put a sentence's meaning in one place; a plain encoder has not, so the defaults here -- the second-to-last layer, mean pooling -- are the ones that transfer best as frozen features (the last layer is specialized to predicting masked words; [CLS] means little in a model never trained to use it).

Long sentences are windowed with overlap and their windows averaged, never cut; sentences from many texts are batched together for throughput; a GPU that runs out of memory halves the batch and carries on. A text with no sentence gets blank cells, not zeros.

extract_transformer_embeddings

extract_transformer_embeddings(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    on_progress=None,
    verbose=True,
    encoding="utf-8-sig",
    delimiter=",",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    pass_through_cols=None,
    model_name_or_path=DEFAULT_ENCODER,
    layers="second_to_last",
    pooling="mean",
    sentence_weighting="equal",
    max_length=512,
    batch_size=32,
    device="auto",
    precision="auto",
    normalize_l2=False,
    rounding=6
)

Embed every text with a transformer encoder: sentence vectors, averaged.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The same input contract as every other text analyzer: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
txt_dir	Optional[PathLike]	The same input contract as every other text analyzer: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
analysis_csv	Optional[PathLike]	The same input contract as every other text analyzer: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
gathered_csv	Optional[PathLike]	The same input contract as every other text analyzer: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
out_features_csv	str or Path	Default ./features/transformer_embeddings.csv.	None
overwrite_existing	bool	If False and the output exists, return it untouched.	False
workers	int	Parallel processes for reading documents during the gather, and the CPU threads torch may use. 0 means automatic.	0
text_cols	sequence of str	When gathering from a CSV, the column(s) holding the text.	("text",)
id_cols	sequence of str	Columns that identify each row when gathering from a CSV.	None
mode	('concat', 'separate')	With several text columns: join them into one text per row, or treat each as its own text.	"concat"
group_by	sequence of str	Columns to combine rows by before analyzing (one text per group).	None
pattern	str	Which files to read when gathering from a folder of documents.	every document type
pass_through_cols	sequence of str	Columns of the source carried into the output beside text_id.	None
model_name_or_path	str	A Hugging Face encoder name (distilroberta-base, sentence-transformers/all-MiniLM-L6-v2, roberta-base, bert-base-uncased, microsoft/deberta-v3-base), a checkpoint folder, a Taters text encoder file (adapted to your corpus), or a Taters fine-tuned predictor file (its encoder is used). Downloaded on first use.	DEFAULT_ENCODER
layers	str	Which hidden layers become the token vectors: second_to_last (the recommended reading of an encoder that was not fine-tuned: the last layer is specialized to predicting masked words), last, last4_mean, last4_concat (four times the width), or a list like -1,-2 (averaged).	"second_to_last"
pooling	('mean', 'cls', 'max')	How a sentence's token vectors become one: the mean over its real tokens, the [CLS] position, or the element-wise maximum.	"mean"
sentence_weighting	('equal', 'tokens')	How a text's sentence vectors become one: each sentence equally (what the sentence-transformers step does) or weighted by length in tokens.	"equal"
max_length	int	The most tokens the encoder reads at once. A longer sentence is windowed with overlap and its windows averaged, never cut.	512
batch_size	int	Sentences per forward pass; halved automatically if the GPU runs out of memory.	32
device	('auto', 'cuda', 'cpu')	Where the encoder runs.	"auto"
precision	('auto', 'fp32', 'fp16')	Half precision on a GPU (auto), always full, or always half.	"auto"
normalize_l2	bool	Scale each text's vector to unit length, for cosine comparisons.	False
rounding	int	Decimal places written.	6

Returns:

Type	Description
Path	out_features_csv: text_id[, pass-through], token_count, sentence_count, e_1..e_d.

Source code in src\taters\text\transformer_embeddings.py

@records_settings(binding=TEXT_INPUT, grain=TEXT_GRAIN,
                  outputs=("out_features_csv",),
                  bookkeeping=("token_count", "sentence_count"))
def extract_transformer_embeddings(
    *,
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    out_features_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    workers: int = 0,
    on_progress: Optional[Callable[..., None]] = None,
    verbose: bool = True,
    encoding: str = "utf-8-sig",
    delimiter: str = ",",
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,
    pass_through_cols: Optional[Sequence[str]] = None,

    # ----- the encoder and how it is read -----
    model_name_or_path: str = DEFAULT_ENCODER,
    layers: str = "second_to_last",
    pooling: Literal["mean", "cls", "max"] = "mean",
    sentence_weighting: Literal["equal", "tokens"] = "equal",
    max_length: int = 512,
    batch_size: int = 32,
    device: Literal["auto", "cuda", "cpu"] = "auto",
    precision: Literal["auto", "fp32", "fp16"] = "auto",
    normalize_l2: bool = False,
    rounding: int = 6,
) -> Path:
    """
    Embed every text with a transformer encoder: sentence vectors, averaged.

    Parameters
    ----------
    csv_path, txt_dir, analysis_csv, gathered_csv
        The same input contract as every other text analyzer: a spreadsheet
        of texts, a folder of documents, or a prebuilt analysis-ready CSV.
    out_features_csv : str or Path, optional
        Default ``./features/transformer_embeddings.csv``.
    overwrite_existing : bool, default False
        If False and the output exists, return it untouched.
    workers : int, default 0
        Parallel processes for reading documents during the gather, and the
        CPU threads torch may use. 0 means automatic.
    text_cols : sequence of str, default ("text",)
        When gathering from a CSV, the column(s) holding the text.
    id_cols : sequence of str, optional
        Columns that identify each row when gathering from a CSV.
    mode : {"concat", "separate"}, default "concat"
        With several text columns: join them into one text per row, or
        treat each as its own text.
    group_by : sequence of str, optional
        Columns to combine rows by before analyzing (one text per group).
    pattern : str, default every document type
        Which files to read when gathering from a folder of documents.
    pass_through_cols : sequence of str, optional
        Columns of the source carried into the output beside ``text_id``.
    model_name_or_path : str
        A Hugging Face encoder name (``distilroberta-base``,
        ``sentence-transformers/all-MiniLM-L6-v2``, ``roberta-base``,
        ``bert-base-uncased``, ``microsoft/deberta-v3-base``), a checkpoint
        folder, a Taters text encoder file (adapted to your corpus), or a
        Taters fine-tuned predictor file (its encoder is used). Downloaded
        on first use.
    layers : str, default "second_to_last"
        Which hidden layers become the token vectors: ``second_to_last``
        (the recommended reading of an encoder that was not fine-tuned: the
        last layer is specialized to predicting masked words), ``last``,
        ``last4_mean``, ``last4_concat`` (four times the width), or a list
        like ``-1,-2`` (averaged).
    pooling : {"mean", "cls", "max"}, default "mean"
        How a sentence's token vectors become one: the mean over its real
        tokens, the ``[CLS]`` position, or the element-wise maximum.
    sentence_weighting : {"equal", "tokens"}, default "equal"
        How a text's sentence vectors become one: each sentence equally
        (what the sentence-transformers step does) or weighted by length in
        tokens.
    max_length : int, default 512
        The most tokens the encoder reads at once. A longer sentence is
        windowed with overlap and its windows averaged, never cut.
    batch_size : int, default 32
        Sentences per forward pass; halved automatically if the GPU runs out
        of memory.
    device : {"auto", "cuda", "cpu"}, default "auto"
        Where the encoder runs.
    precision : {"auto", "fp32", "fp16"}, default "auto"
        Half precision on a GPU (auto), always full, or always half.
    normalize_l2 : bool, default False
        Scale each text's vector to unit length, for cosine comparisons.
    rounding : int, default 6
        Decimal places written.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id[, pass-through], token_count,
        sentence_count, e_1..e_d``.
    """
    import numpy as np

    missing = torch_missing_reason()
    if missing:
        raise ImportError(missing)
    if sentence_weighting not in ("equal", "tokens"):
        raise ValueError("sentence_weighting must be 'equal' or 'tokens', not "
                         f"{sentence_weighting!r}")
    from ..helpers.nltk_data import ensure_punkt

    ensure_punkt(verbose=verbose)

    from ..helpers.row_map import resolve_passthrough_columns

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers, carry_cols=list(pass_through_cols or []) or None,
        verbose=verbose)
    out_features_csv = Path(out_features_csv) if out_features_csv else \
        Path.cwd() / "features" / "transformer_embeddings.csv"
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)
    if not overwrite_existing and out_features_csv.is_file():
        if verbose:
            print("Transformer embeddings output already exists; returning existing file.")
        return out_features_csv

    announce(on_progress, f"loading the encoder ({model_name_or_path})")
    set_threads(workers)
    model, tokenizer, resolved, device_name, reason = load_encoder(
        model_name_or_path, device=device, verbose=verbose)
    with analysis_ready.open("r", newline="", encoding=encoding) as rf:
        header_fields = csv.DictReader(rf, delimiter=delimiter).fieldnames or []
    passthrough = resolve_passthrough_columns(
        header_fields, pass_through_cols=pass_through_cols, id_cols=id_cols,
        group_by=group_by, analysis_ready=analysis_ready)

    total = count_rows(analysis_ready, on_progress=on_progress)
    # which device we ended up on, said where it stays put: the ticks below
    # carry no message, so this is the label somebody watches for the whole
    # encode. it used to be a print behind `verbose`, which the app never
    # sets -- so an hour-long run on the CPU looked exactly like a fast one
    # on the card, and the only way to find out was to watch the GPU meter
    announce(on_progress, device_note("encoding", device_name, reason))
    ticker = Ticker(on_progress, total)
    dim: Optional[int] = None

    def encode(sentences: List[str]):
        return encode_sentences(
            model, tokenizer, sentences, layers=layers, pooling=pooling,
            max_length=max_length, batch_size=batch_size,
            device_name=device_name, precision=precision, verbose=verbose)

    def flush(pending, writer):
        """Encode the buffered texts' sentences in one go and write a row
        per text -- the batch spans texts, the averaging never does."""
        nonlocal dim
        flat = [s for _row, sents in pending for s in sents]
        vectors, counts = encode(flat)
        if dim is None:
            dim = int(vectors.shape[1]) if vectors.shape[0] else int(
                model.config.hidden_size) * (4 if layers == "last4_concat" else 1)
        at = 0
        for row, sents in pending:
            k = len(sents)
            vecs, cnts = vectors[at:at + k], counts[at:at + k]
            at += k
            doc = _doc_vector(vecs, cnts, sentence_weighting)
            cells: List[object] = [row.get("text_id", "")]
            cells += [row.get(c, "") for c in passthrough]
            cells += [int(sum(cnts)), k]
            if doc is None:
                cells += [None] * dim
            else:
                if normalize_l2:
                    n = float(np.linalg.norm(doc))
                    if n > 1e-12:
                        doc = doc / n
                cells += round_cells(doc, rounding)
            writer.writerow(cells)
            ticker.tick()

    with atomic_write(out_features_csv, newline="", encoding=encoding) as out, \
            analysis_ready.open("r", newline="", encoding=encoding) as rf:
        reader = csv.DictReader(rf, delimiter=delimiter)
        writer = csv.writer(out)
        # we can't write the header until we know the embedding width, and we
        # don't know that until the first forward pass. so, we buffer rows until
        # the first flush and write the header then
        header_written = False
        pending: list = []
        buffered = 0
        for row in reader:
            sents = split_sentences(row.get("text") or "")
            pending.append((row, sents))
            buffered += len(sents)
            if buffered >= _SENTENCES_PER_CALL:
                if not header_written:
                    _probe_dim(pending, encode, model, layers)
                    dim = _probe_dim.dim
                    writer.writerow(_header(passthrough, dim))
                    header_written = True
                flush(pending, writer)
                pending, buffered = [], 0
        if not header_written:
            _probe_dim(pending, encode, model, layers)
            dim = _probe_dim.dim
            writer.writerow(_header(passthrough, dim))
        if pending:
            flush(pending, writer)
    if verbose:
        print(f"[transformer_embeddings] {total} text(s) embedded with {resolved.label} "
              f"(layers={layers}, pooling={pooling}) on {device_name} -> {out_features_csv}")
    return out_features_csv

taters.text.word_vectors

Word vectors: train them on your own texts, bring them from elsewhere, and turn them into features.

A word vector model learns, from nothing but co-occurrence, that grave sits near death and far from picnic. Trained on the corpus under study it captures how these writers use words -- what "home" means in a bereavement forum is not what it means in a real-estate listing -- and a pre-trained set (GloVe, word2vec, fastText) brings a general-purpose sense of the language to a corpus too small to train on. Either way the model is a matrix: one row per word, one column per dimension.

Three things are done with it here:

• The mean vector per text -- wv_1 .. wv_k -- the classic bag-of- vectors representation, a compact numeric fingerprint that a ridge or classifier can learn from.
• Similarity to concepts -- sim_mydict__death, sim_mydict__work -- the cosine between a text's vector and the weighted mean vector of a category of a LIWC-22 dictionary (.dic, .dicx, .csv: one column per category, wildcards and phrases as LIWC reads them, cells as weights), so a hypothesis ("these texts dwell on mortality") becomes one column per category. See :mod:taters.text._concept_dicts.
• Nearest neighbors of chosen words, as a table and as word clouds, so the model can be inspected and reported: the neighbors are the evidence that it learned what you think it learned.

Two commitments shape the module:

• A fitted model is a reusable instrument. The manifest (.json) records the vocabulary, the text settings that produced it, how it was trained and how it should be applied; the matrix travels beside it as <stem>.npy (a payload, in the library's terms -- moved, renamed and deleted with the manifest). Applying the model to its own training texts reproduces the training features exactly.
• Memory-safe on an ordinary laptop. The matrix is memory-mapped, never copied whole; nearest neighbors are computed in row chunks; training streams the tokenized corpus from a scratch file (gensim keeps only the vocabulary and the matrix in memory). gensim is needed only to train and to read the binary word2vec/fastText formats; applying, describing and importing text formats need numpy alone.

fastText's subword vectors are not kept (the bucket matrix is hundreds of megabytes and mostly noise for feature extraction); a word outside the vocabulary is skipped for both families, and the in_vocab_count column says how many words each text lost that way.

apply_word_vectors

apply_word_vectors(
    *,
    model_json,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    on_progress=None,
    verbose=True,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    device="auto",
    rounding=4,
    weighting=None,
    normalize_words=None,
    concept_dicts=None
)

Score new texts with a saved word-vector model.

The model's own settings (its apply block: weighting, word normalization, concepts) are used unless the call gives its own, so a pipeline that applies a model gets the settings the model was saved with -- and changed in Settings afterwards -- without knowing them.

Parameters:

Name	Type	Description	Default
model_json	PathLike	A word-vector manifest written by :func:train_word_vectors or :func:import_word_vectors; its matrix is read from beside it, or from the library when the manifest traveled alone.	required
csv_path	Optional[PathLike]	The same input contract as the other text analyzers.	None
txt_dir	Optional[PathLike]	The same input contract as the other text analyzers.	None
analysis_csv	Optional[PathLike]	The same input contract as the other text analyzers.	None
gathered_csv	Optional[PathLike]	The same input contract as the other text analyzers.	None
out_features_csv	str or Path	Default ./features/word_vectors_applied.csv.	None
overwrite_existing	bool	If False and the output files exist, return them untouched.	False
workers	int	Parallel processes for reading and tokenizing texts. 0 means automatic: three-quarters of the logical cores.	0
text_cols	sequence of str	When gathering from a CSV, the column(s) holding the text.	("text",)
id_cols	sequence of str	Columns that identify each row when gathering from a CSV.	None
mode	('concat', 'separate')	With several text columns: join them into one text per row, or treat each as its own text.	"concat"
group_by	sequence of str	Columns to combine rows by before analyzing (one text per group).	None
pattern	str	Which files to read when gathering from a folder of documents.	every document type
device	('auto', 'cuda', 'cpu')	Where Stanza runs, if the stanza engine is used -- a runtime choice, deliberately not stored in the model.	"auto"
rounding	int	Decimal places written.	4
weighting	Optional[Literal['tokens', 'types', 'sif']]	Overrides for the model's own apply settings; see :func:train_word_vectors. concept_dicts are dictionary files; left out, the dictionaries stored in the model apply.	None
normalize_words	Optional[Literal['tokens', 'types', 'sif']]	Overrides for the model's own apply settings; see :func:train_word_vectors. concept_dicts are dictionary files; left out, the dictionaries stored in the model apply.	None
concept_dicts	Optional[Literal['tokens', 'types', 'sif']]	Overrides for the model's own apply settings; see :func:train_word_vectors. concept_dicts are dictionary files; left out, the dictionaries stored in the model apply.	None

Returns:

Type	Description
Path	out_features_csv: text_id, token_count, in_vocab_count, wv_1..wv_k, sim_<dictionary>__<category>....

Source code in src\taters\text\word_vectors.py

@records_settings(
    binding=TEXT_INPUT, grain=TEXT_GRAIN, assets={"model_json": None},
    outputs=("out_features_csv",), bookkeeping=("token_count", "in_vocab_count"))
def apply_word_vectors(
    *,
    model_json: PathLike,
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    out_features_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    workers: int = 0,
    on_progress: Optional[Callable[..., None]] = None,
    verbose: bool = True,
    encoding: str = "utf-8-sig",
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,
    device: str = "auto",
    rounding: int = 4,
    weighting: Optional[Literal["tokens", "types", "sif"]] = None,
    normalize_words: Optional[bool] = None,
    concept_dicts: Optional[Sequence[PathLike]] = None,
) -> Path:
    """
    Score new texts with a saved word-vector model.

    The model's own settings (its ``apply`` block: weighting, word
    normalization, concepts) are used unless the call gives its own, so a
    pipeline that applies a model gets the settings the model was saved
    with -- and changed in Settings afterwards -- without knowing them.

    Parameters
    ----------
    model_json
        A word-vector manifest written by :func:`train_word_vectors` or
        :func:`import_word_vectors`; its matrix is read from beside it, or
        from the library when the manifest traveled alone.
    csv_path, txt_dir, analysis_csv, gathered_csv
        The same input contract as the other text analyzers.
    out_features_csv : str or Path, optional
        Default ``./features/word_vectors_applied.csv``.
    overwrite_existing : bool, default False
        If False and the output files exist, return them untouched.
    workers : int, default 0
        Parallel processes for reading and tokenizing texts. 0 means
        automatic: three-quarters of the logical cores.
    text_cols : sequence of str, default ("text",)
        When gathering from a CSV, the column(s) holding the text.
    id_cols : sequence of str, optional
        Columns that identify each row when gathering from a CSV.
    mode : {"concat", "separate"}, default "concat"
        With several text columns: join them into one text per row, or
        treat each as its own text.
    group_by : sequence of str, optional
        Columns to combine rows by before analyzing (one text per group).
    pattern : str, default every document type
        Which files to read when gathering from a folder of documents.
    device : {"auto", "cuda", "cpu"}, default "auto"
        Where Stanza runs, if the stanza engine is used -- a runtime choice,
        deliberately not stored in the model.
    rounding : int, default 4
        Decimal places written.
    weighting, normalize_words, concept_dicts
        Overrides for the model's own apply settings; see
        :func:`train_word_vectors`. ``concept_dicts`` are dictionary files;
        left out, the dictionaries stored in the model apply.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id, token_count, in_vocab_count,
        wv_1..wv_k, sim_<dictionary>__<category>...``.
    """
    model = _load_model(one_model_path(model_json))
    weighting, normalize_words, dicts = _settings_of(
        model, weighting, normalize_words, concept_dicts)
    text = model.text_settings
    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)
    out_features_csv = Path(out_features_csv) if out_features_csv else \
        Path.cwd() / "features" / "word_vectors_applied.csv"
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)
    if not overwrite_existing and out_features_csv.is_file():
        if verbose:
            print("Word-vector features already exist; returning existing file.")
        return out_features_csv

    if text["engine"] == "stanza":
        announce(on_progress, "loading the stanza pipeline (first use downloads its model)")
    stream = _stream_for(text, device)
    lowercase = bool(text.get("lowercase", True))
    concept_columns, concept_vecs, concept_rows = _concept_vectors(
        model, dicts, stream, lowercase)
    from ..helpers.row_map import map_text_rows
    from .ngram_prep import pooled_text_workers

    with atomic_write(out_features_csv, mode="w", newline="", encoding=encoding) as out:
        writer = csv.writer(out)
        writer.writerow(_header(model, concept_columns))
        for row, tokens in map_text_rows(
                analysis_ready, encoding=encoding,
                workers=lambda n_rows: pooled_text_workers(
                    workers, n_rows, engine=str(text["engine"]),
                    tokenizer=str(text["tokenizer"]),
                    lemmatize=bool(text["lemmatize"]), pos_tagged=False),
                message="scoring word vectors", on_progress=on_progress,
                inline_fn=lambda pair: _tokens_of(stream, lowercase, pair[1]),
                pool_fn=_tokens_in_worker, initializer=_init_token_worker,
                initargs=(_stream_args(text, device), lowercase)):
            clean = [t.replace(" ", "_") for t in tokens if t.strip()]
            writer.writerow([row.get("text_id", ""), *_feature_row(
                model, clean, weighting=weighting, normalize_words=normalize_words,
                concept_vectors=concept_vecs, rounding=rounding)])
    if verbose:
        for row in concept_rows:
            if row["missed"]:
                print(f"[word_vectors] note: {row['column']}: {len(row['missed'])} of "
                      f"{row['n_terms']} term(s) not in the vocabulary")
    return out_features_csv

describe_word_vectors

describe_word_vectors(
    model_json,
    out_neighbors_csv=None,
    *,
    probes="",
    top_neighbors=20,
    device="auto",
    rounding=4,
    encoding="utf-8-sig",
    overwrite_existing=False,
    verbose=True,
    on_progress=None
)

Write the nearest-neighbors table of a saved model.

probes are comma-separated words; empty means the model's most frequent words. Every concept category the model carries is listed too, as the words closest to its vector. The table has one row per (probe, neighbor): probe, rank, word, similarity; a probe not in the vocabulary gets one row with blanks, so its absence is visible.

Source code in src\taters\text\word_vectors.py

def describe_word_vectors(
    model_json: PathLike,
    out_neighbors_csv: Optional[PathLike] = None,
    *,
    probes: str = "",
    top_neighbors: int = 20,
    device: str = "auto",
    rounding: int = 4,
    encoding: str = "utf-8-sig",
    overwrite_existing: bool = False,
    verbose: bool = True,
    on_progress: Optional[Callable[..., None]] = None,
) -> Path:
    """
    Write the nearest-neighbors table of a saved model.

    ``probes`` are comma-separated words; empty means the model's most
    frequent words. Every concept category the model carries is listed
    too, as the words closest to its vector. The table has one row
    per (probe, neighbor): ``probe, rank, word, similarity``; a probe not
    in the vocabulary gets one row with blanks, so its absence is visible.
    """
    path = one_model_path(model_json)
    model = _load_model(path)
    out = Path(out_neighbors_csv) if out_neighbors_csv else \
        path.with_name(path.stem + "_neighbors.csv")
    if out.is_file() and not overwrite_existing:
        if verbose:
            print("Neighbors table already exists; returning existing file.")
        return out
    announce(on_progress, "finding nearest neighbors")
    stream = _stream_for(model.text_settings, device)
    dicts = concept_dicts_from_json((model.doc.get("apply") or {}).get("concept_dicts") or [])
    words = _probe_words(probes, list(model.index),
                         model.counts if model.counts is not None else None)
    neighbors = _neighbors(model, words, stream, top_n=int(top_neighbors))
    if dicts:
        columns, vecs, _rows = _concept_vectors(model, dicts, stream,
                                                bool(model.text_settings.get("lowercase", True)))
        neighbors.update(_neighbors_of_vectors(model, columns, vecs,
                                                 top_n=int(top_neighbors)))
    _write_neighbors(out, neighbors, encoding=encoding, rounding=rounding)
    if verbose:
        print(f"[word_vectors] neighbors of {len(words)} word(s) -> {out}")
    return out

import_word_vectors

import_word_vectors(
    vectors_path,
    out_model_json,
    *,
    format="auto",
    max_vocab=200000,
    name=None,
    concept_dicts=(),
    weighting="tokens",
    normalize_words=False,
    lemmatize=False,
    keep_punctuation=False,
    engine="nltk",
    tokenizer="potts",
    stanza_lang="en",
    probes="",
    top_neighbors=20,
    out_report_md=None,
    overwrite_existing=False,
    encoding="utf-8",
    verbose=True,
    on_progress=None
)

Bring pre-trained vectors (GloVe, word2vec, fastText) in as a model.

Parameters:

Name	Type	Description	Default
vectors_path	PathLike	The vectors file: GloVe text (word v1 v2 ...), word2vec text (a V k header line first) or binary, or a fastText .bin. The binary formats need gensim (pip install "taters[vectors]").	required
out_model_json	PathLike	Where the manifest goes; the matrix lands beside it as .npy.	required
format	Literal['auto', 'glove', 'word2vec_text', 'word2vec_bin', 'fasttext_bin']	auto sniffs the file; name it when the sniff is wrong.	'auto'
max_vocab	int	The most words kept, from the top of the file (these files list words by frequency). 200,000 x 300 floats is 240 MB, the most an ordinary laptop should be asked to hold. Case is folded, keeping the first occurrence of a word, because every tokenizer here lower-cases text: a vocabulary with both Cat and cat would never see the first.	200000
lemmatize	bool	How texts will be tokenized when the model is applied. Off for lemmatizing by default: pre-trained vectors were learned on inflected words, and "ran" is in the vocabulary while "run" alone would miss it.	False
keep_punctuation	bool	How texts will be tokenized when the model is applied. Off for lemmatizing by default: pre-trained vectors were learned on inflected words, and "ran" is in the vocabulary while "run" alone would miss it.	False
engine	bool	How texts will be tokenized when the model is applied. Off for lemmatizing by default: pre-trained vectors were learned on inflected words, and "ran" is in the vocabulary while "run" alone would miss it.	False
tokenizer	bool	How texts will be tokenized when the model is applied. Off for lemmatizing by default: pre-trained vectors were learned on inflected words, and "ran" is in the vocabulary while "run" alone would miss it.	False
stanza_lang	bool	How texts will be tokenized when the model is applied. Off for lemmatizing by default: pre-trained vectors were learned on inflected words, and "ran" is in the vocabulary while "run" alone would miss it.	False
name	Optional[str]	As for :func:train_word_vectors. sif weighting is not offered: imported vectors carry no counts.	None
concept_dicts	Optional[str]	As for :func:train_word_vectors. sif weighting is not offered: imported vectors carry no counts.	None
weighting	Optional[str]	As for :func:train_word_vectors. sif weighting is not offered: imported vectors carry no counts.	None
normalize_words	Optional[str]	As for :func:train_word_vectors. sif weighting is not offered: imported vectors carry no counts.	None
probes	Optional[str]	As for :func:train_word_vectors. sif weighting is not offered: imported vectors carry no counts.	None
top_neighbors	Optional[str]	As for :func:train_word_vectors. sif weighting is not offered: imported vectors carry no counts.	None

Returns:

Type	Description
Path	out_model_json.

Source code in src\taters\text\word_vectors.py

def import_word_vectors(
    vectors_path: PathLike,
    out_model_json: PathLike,
    *,
    format: Literal["auto", "glove", "word2vec_text", "word2vec_bin",
                    "fasttext_bin"] = "auto",
    max_vocab: int = 200000,
    name: Optional[str] = None,
    concept_dicts: Sequence[PathLike] = (),
    weighting: Literal["tokens", "types"] = "tokens",
    normalize_words: bool = False,
    lemmatize: bool = False,
    keep_punctuation: bool = False,
    engine: Literal["nltk", "stanza"] = "nltk",
    tokenizer: Literal["potts", "stanza"] = "potts",
    stanza_lang: str = "en",
    probes: str = "",
    top_neighbors: int = 20,
    out_report_md: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    encoding: str = "utf-8",
    verbose: bool = True,
    on_progress: Optional[Callable[..., None]] = None,
) -> Path:
    """
    Bring pre-trained vectors (GloVe, word2vec, fastText) in as a model.

    Parameters
    ----------
    vectors_path
        The vectors file: GloVe text (``word v1 v2 ...``), word2vec text
        (a ``V k`` header line first) or binary, or a fastText ``.bin``.
        The binary formats need gensim (``pip install "taters[vectors]"``).
    out_model_json
        Where the manifest goes; the matrix lands beside it as ``.npy``.
    format
        ``auto`` sniffs the file; name it when the sniff is wrong.
    max_vocab
        The most words kept, from the top of the file (these files list
        words by frequency). 200,000 x 300 floats is 240 MB, the most an
        ordinary laptop should be asked to hold. Case is folded, keeping
        the first occurrence of a word, because every tokenizer here
        lower-cases text: a vocabulary with both *Cat* and *cat* would
        never see the first.
    lemmatize, keep_punctuation, engine, tokenizer, stanza_lang
        How texts will be tokenized when the model is applied. Off for
        lemmatizing by default: pre-trained vectors were learned on
        inflected words, and "ran" is in the vocabulary while "run" alone
        would miss it.
    name, concept_dicts, weighting, normalize_words, probes, top_neighbors
        As for :func:`train_word_vectors`. ``sif`` weighting is not offered:
        imported vectors carry no counts.

    Returns
    -------
    Path
        ``out_model_json``.
    """
    import numpy as np

    started = time.time()
    src = Path(vectors_path)
    if not src.is_file():
        raise FileNotFoundError(f"vectors file not found: {src}")
    out_model_json = Path(out_model_json)
    if out_model_json.is_file() and not overwrite_existing:
        if verbose:
            print("Word-vector model already exists; returning existing file.")
        return out_model_json
    if format not in VECTOR_FORMATS:
        raise ValueError(f"format must be one of {', '.join(VECTOR_FORMATS)}, "
                         f"not {format!r}")
    if weighting not in ("tokens", "types"):
        raise ValueError("imported vectors carry no word counts, so weighting "
                         "must be 'tokens' or 'types'")
    dicts = load_concept_dicts(list(concept_dicts))
    lowercase = True
    fmt = _sniff_format(src) if format == "auto" else format
    announce(on_progress, f"reading {src.name} as {fmt}")
    if fmt in ("glove", "word2vec_text"):
        words, matrix = _read_text_vectors(
            src, header=(fmt == "word2vec_text"), max_vocab=max_vocab,
            lowercase=lowercase, encoding=encoding)
    else:
        missing = _gensim_missing()
        if missing:
            raise ImportError(missing.replace("Training word vectors", "Reading binary vectors"))
        if fmt == "fasttext_bin":
            from gensim.models.fasttext import load_facebook_vectors

            kv = load_facebook_vectors(str(src))
        else:
            from gensim.models import KeyedVectors

            kv = KeyedVectors.load_word2vec_format(str(src), binary=True,
                                                   limit=int(max_vocab))
        words, keep = [], []
        seen = set()
        for i, w in enumerate(kv.index_to_key[:int(max_vocab)]):
            w = w.lower() if lowercase else w
            if w in seen:
                continue
            seen.add(w)
            words.append(w)
            keep.append(i)
        matrix = np.asarray(kv.vectors[keep], dtype=np.float32)
    dim = int(matrix.shape[1])
    out_model_json.parent.mkdir(parents=True, exist_ok=True)
    weights = out_model_json.with_suffix(".npy")
    with atomic_write(weights, mode="wb") as fh:
        np.save(fh, matrix)
    text_settings = {"lemmatize": bool(lemmatize), "pos_tagged": False,
                     "engine": engine, "tokenizer": tokenizer,
                     "stanza_lang": stanza_lang,
                     "keep_punctuation": bool(keep_punctuation),
                     "lowercase": bool(lowercase)}
    training = {"source": "imported", "format": fmt, "file": src.name,
                "file_size": src.stat().st_size, "vocabulary_size": len(words),
                "max_vocab": int(max_vocab), "wall_seconds": None}
    _write_manifest(out_model_json, weights, name=str(name or out_model_json.stem),
                    text=text_settings, vocabulary=words, counts=None, dim=dim,
                    training=training, concept_dicts=dicts,
                    weighting=weighting, normalize_words=normalize_words)
    model = _load_model(out_model_json)
    stream = _stream_for(text_settings, "cpu")
    # this refuses any category the vectors don't know about, before we save
    # anything as a model that somebody will try to use
    concept_columns, concept_vecs, concept_rows = _concept_vectors(
        model, dicts, stream, bool(lowercase))
    probe_words = _probe_words(probes, words, None)
    neighbors = _neighbors(model, probe_words, stream, top_n=int(top_neighbors))
    neighbors.update(_neighbors_of_vectors(model, concept_columns, concept_vecs,
                                             top_n=int(top_neighbors)))
    neighbors_csv = out_model_json.with_name(out_model_json.stem + "_neighbors.csv")
    _write_neighbors(neighbors_csv, neighbors, encoding="utf-8-sig", rounding=4)
    training["wall_seconds"] = round(time.time() - started, 1)
    doc = json.loads(out_model_json.read_text(encoding="utf-8"))
    doc["training"] = training
    with atomic_write(out_model_json, mode="w", encoding="utf-8") as fh:
        json.dump(doc, fh, indent=1)
    report = Path(out_report_md) if out_report_md else \
        out_model_json.with_name(out_model_json.stem + "_report.md")
    _write_report(report, doc, out_model_json, weights, neighbors, concept_rows,
                  features_csv=None, neighbors_csv=neighbors_csv, model=model)
    if verbose:
        print(f"[word_vectors] imported {len(words)} words x {dim} dims from "
              f"{src.name} -> {out_model_json}")
    return out_model_json

nearest_neighbors

nearest_neighbors(
    model_json, words, *, top_n=20, device="auto"
)

The top_n most similar vocabulary words to each word, by cosine.

A word is looked up as the model tokenizes it (lower-cased, lemmatized if the model was); a word not in the vocabulary maps to an empty list. The probe itself is never among its own neighbors. Computed in row chunks, so a large model is searched without a second copy in memory.

Source code in src\taters\text\word_vectors.py

def nearest_neighbors(model_json: PathLike, words: Sequence[str], *,
                       top_n: int = 20, device: str = "auto"
                       ) -> Dict[str, List[Tuple[str, float]]]:
    """
    The ``top_n`` most similar vocabulary words to each word, by cosine.

    A word is looked up as the model tokenizes it (lower-cased, lemmatized
    if the model was); a word not in the vocabulary maps to an empty list.
    The probe itself is never among its own neighbors. Computed in row
    chunks, so a large model is searched without a second copy in memory.
    """
    model = _load_model(one_model_path(model_json))
    stream = _stream_for(model.text_settings, device)
    return _neighbors(model, words, stream, top_n=top_n)

train_word_vectors

train_word_vectors(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    out_features_csv=None,
    out_model_json=None,
    out_neighbors_csv=None,
    out_report_md=None,
    overwrite_existing=False,
    workers=0,
    on_progress=None,
    verbose=True,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    device="auto",
    lemmatize=False,
    keep_punctuation=False,
    engine="nltk",
    tokenizer="potts",
    stanza_lang="en",
    name=None,
    family="word2vec",
    algorithm="skipgram",
    vector_size=100,
    window=5,
    min_count=5,
    epochs=5,
    negative=5,
    seed=42,
    reproducible=False,
    concept_dicts=(),
    weighting="tokens",
    normalize_words=False,
    probes="",
    top_neighbors=20,
    rounding=4
)

Train word vectors on the texts, save the model, and write its features.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The same input contract as every other text analyzer: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
txt_dir	Optional[PathLike]	The same input contract as every other text analyzer: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
analysis_csv	Optional[PathLike]	The same input contract as every other text analyzer: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
gathered_csv	Optional[PathLike]	The same input contract as every other text analyzer: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
out_features_csv	str or Path	The features -- one row per text -- default ./features/word_vectors.csv.	None
overwrite_existing	bool	If False and the output files exist, return them untouched.	False
workers	int	Parallel processes for reading and tokenizing texts. 0 means automatic: three-quarters of the logical cores.	0
text_cols	sequence of str	When gathering from a CSV, the column(s) holding the text.	("text",)
id_cols	sequence of str	Columns that identify each row when gathering from a CSV.	None
mode	('concat', 'separate')	With several text columns: join them into one text per row, or treat each as its own text.	"concat"
group_by	sequence of str	Columns to combine rows by before analyzing (one text per group).	None
pattern	str	Which files to read when gathering from a folder of documents.	every document type
device	('auto', 'cuda', 'cpu')	Where Stanza runs, if the stanza engine is used -- a runtime choice, deliberately not stored in the model.	"auto"
out_model_json	str or Path	The model manifest; its matrix lands beside it as <stem>.npy. Default <features folder>/models/word_vectors.json.	None
out_neighbors_csv	str or Path	The nearest-neighbors table and the training report, default beside the model as <stem>_neighbors.csv and <stem>_report.md.	None
out_report_md	str or Path	The nearest-neighbors table and the training report, default beside the model as <stem>_neighbors.csv and <stem>_report.md.	None
lemmatize	bool	How text becomes words, recorded in the model so new texts are read the same way. Text is always lower-cased (every tokenizer here does it): Death and death are one word to a study of meaning, and a vocabulary that kept both would spend its counts twice.	False
keep_punctuation	bool	How text becomes words, recorded in the model so new texts are read the same way. Text is always lower-cased (every tokenizer here does it): Death and death are one word to a study of meaning, and a vocabulary that kept both would spend its counts twice.	False
engine	bool	How text becomes words, recorded in the model so new texts are read the same way. Text is always lower-cased (every tokenizer here does it): Death and death are one word to a study of meaning, and a vocabulary that kept both would spend its counts twice.	False
tokenizer	bool	How text becomes words, recorded in the model so new texts are read the same way. Text is always lower-cased (every tokenizer here does it): Death and death are one word to a study of meaning, and a vocabulary that kept both would spend its counts twice.	False
stanza_lang	bool	How text becomes words, recorded in the model so new texts are read the same way. Text is always lower-cased (every tokenizer here does it): Death and death are one word to a study of meaning, and a vocabulary that kept both would spend its counts twice.	False
name	str	The model's name in menus; default the manifest's file stem.	None
family	('word2vec', 'fasttext')	word2vec learns a vector per word; fastText also learns from character n-grams during training, which helps with rare and misspelt words, though only whole-word vectors are kept here.	"word2vec"
algorithm	('skipgram', 'cbow')	Skip-gram predicts context from a word and does better on small corpora and rare words; CBOW is faster and slightly better on very large ones.	"skipgram"
vector_size	int	The usual: dimensions; context words either side; the fewest occurrences a word needs to get a vector; passes over the corpus; negative samples per positive.	100
window	int	The usual: dimensions; context words either side; the fewest occurrences a word needs to get a vector; passes over the corpus; negative samples per positive.	100
min_count	int	The usual: dimensions; context words either side; the fewest occurrences a word needs to get a vector; passes over the corpus; negative samples per positive.	100
epochs	int	The usual: dimensions; context words either side; the fewest occurrences a word needs to get a vector; passes over the corpus; negative samples per positive.	100
negative	int	The usual: dimensions; context words either side; the fewest occurrences a word needs to get a vector; passes over the corpus; negative samples per positive.	100
seed	int	The random seed. Training runs in parallel threads and is then reproducible only in distribution; reproducible=True trains on one thread so the same corpus and seed give the same vectors (slower).	42
reproducible	int	The random seed. Training runs in parallel threads and is then reproducible only in distribution; reproducible=True trains on one thread so the same corpus and seed give the same vectors (slower).	42
concept_dicts	sequence of str or Path	LIWC-22 dictionaries (.dic, .dicx, .csv; a folder means every dictionary in it). Every category becomes a sim_<dictionary>__<category> column: the cosine between a text's vector and the category's vector, the weighted mean of its terms' vectors (cells are weights, X = 1; wildcards and phrases as LIWC reads them). The dictionaries' terms and weights are stored in the model, and can be changed later in Settings.	()
weighting	('tokens', 'types', 'sif')	How words are averaged into a text's vector: every occurrence (tokens), each distinct word once (types), or smooth inverse frequency (sif: the commonest words count least).	"tokens"
normalize_words	bool	Scale every word vector to unit length before averaging, so a frequent word with a long vector does not dominate.	False
probes	str	Comma-separated words whose nearest neighbors the report shows; every concept category's neighbors are shown as well. Empty: :data:DEFAULT_PROBES, a dozen common content words, skipping any the model did not learn. Nothing about training depends on this; it only decides which words the clouds are about.	''
top_neighbors	int	Neighbors per probe in the table and the clouds.	20
rounding	int	Decimal places written.	4

Returns:

Type	Description
Path	out_features_csv.

Notes

Nothing survives below min_count on a tiny corpus, and the refusal says so before gensim is asked. The report beside the model has the methods paragraph, the settings, the corpus coverage, the loss per epoch, the neighbors, and the package versions -- what a paper needs.

Source code in src\taters\text\word_vectors.py

@records_settings(
    binding=TEXT_INPUT, grain=TEXT_GRAIN,
    # the concept dictionaries are word lists: their *content* is what the
    # features depend on, so it gets hashed and carried like dict_paths is
    assets={"concept_dicts": "dictionaries"},
    outputs=("out_features_csv", "out_model_json", "out_neighbors_csv",
             "out_report_md"),
    # the vectors are fitted to this corpus, so the honest way to measure them
    # on another is to apply the saved model. retraining on the new texts
    # would give us different dimensions, and a ridge fitted on these ones
    # couldn't be scored at all
    replay=(f"{__name__}:apply_word_vectors", {"model_json": "out_model_json"}),
    bookkeeping=("token_count", "in_vocab_count"))
def train_word_vectors(
    *,
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    out_features_csv: Optional[PathLike] = None,
    out_model_json: Optional[PathLike] = None,
    out_neighbors_csv: Optional[PathLike] = None,
    out_report_md: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    workers: int = 0,
    on_progress: Optional[Callable[..., None]] = None,
    verbose: bool = True,
    encoding: str = "utf-8-sig",
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,
    device: str = "auto",

    # ----- how we read the text (we record these into the model) -----
    lemmatize: bool = False,
    keep_punctuation: bool = False,
    engine: Literal["nltk", "stanza"] = "nltk",
    tokenizer: Literal["potts", "stanza"] = "potts",
    stanza_lang: str = "en",

    # ----- the model -----
    name: Optional[str] = None,
    family: Literal["word2vec", "fasttext"] = "word2vec",
    algorithm: Literal["skipgram", "cbow"] = "skipgram",
    vector_size: int = 100,
    window: int = 5,
    min_count: int = 5,
    epochs: int = 5,
    negative: int = 5,
    seed: int = 42,
    reproducible: bool = False,

    # ----- how we apply it, to this corpus now and to others later -----
    concept_dicts: Sequence[PathLike] = (),
    weighting: Literal["tokens", "types", "sif"] = "tokens",
    normalize_words: bool = False,
    probes: str = "",
    top_neighbors: int = 20,
    rounding: int = 4,
) -> Path:
    """
    Train word vectors on the texts, save the model, and write its features.

    Parameters
    ----------
    csv_path, txt_dir, analysis_csv, gathered_csv
        The same input contract as every other text analyzer: a spreadsheet
        of texts, a folder of documents, or a prebuilt analysis-ready CSV.
    out_features_csv : str or Path, optional
        The features -- one row per text -- default
        ``./features/word_vectors.csv``.
    overwrite_existing : bool, default False
        If False and the output files exist, return them untouched.
    workers : int, default 0
        Parallel processes for reading and tokenizing texts. 0 means
        automatic: three-quarters of the logical cores.
    text_cols : sequence of str, default ("text",)
        When gathering from a CSV, the column(s) holding the text.
    id_cols : sequence of str, optional
        Columns that identify each row when gathering from a CSV.
    mode : {"concat", "separate"}, default "concat"
        With several text columns: join them into one text per row, or
        treat each as its own text.
    group_by : sequence of str, optional
        Columns to combine rows by before analyzing (one text per group).
    pattern : str, default every document type
        Which files to read when gathering from a folder of documents.
    device : {"auto", "cuda", "cpu"}, default "auto"
        Where Stanza runs, if the stanza engine is used -- a runtime choice,
        deliberately not stored in the model.
    out_model_json : str or Path, optional
        The model manifest; its matrix lands beside it as ``<stem>.npy``.
        Default ``<features folder>/models/word_vectors.json``.
    out_neighbors_csv, out_report_md : str or Path, optional
        The nearest-neighbors table and the training report, default
        beside the model as ``<stem>_neighbors.csv`` and ``<stem>_report.md``.
    lemmatize, keep_punctuation, engine, tokenizer, stanza_lang
        How text becomes words, recorded in the model so new texts are read
        the same way. Text is always lower-cased (every tokenizer here does
        it): *Death* and *death* are one word to a study of meaning, and a
        vocabulary that kept both would spend its counts twice.
    name : str, optional
        The model's name in menus; default the manifest's file stem.
    family : {"word2vec", "fasttext"}
        word2vec learns a vector per word; fastText also learns from
        character n-grams during training, which helps with rare and
        misspelt words, though only whole-word vectors are kept here.
    algorithm : {"skipgram", "cbow"}
        Skip-gram predicts context from a word and does better on small
        corpora and rare words; CBOW is faster and slightly better on very
        large ones.
    vector_size, window, min_count, epochs, negative
        The usual: dimensions; context words either side; the fewest
        occurrences a word needs to get a vector; passes over the corpus;
        negative samples per positive.
    seed, reproducible
        The random seed. Training runs in parallel threads and is then
        reproducible only in distribution; ``reproducible=True`` trains on
        one thread so the same corpus and seed give the same vectors (slower).
    concept_dicts : sequence of str or Path
        LIWC-22 dictionaries (``.dic``, ``.dicx``, ``.csv``; a folder means
        every dictionary in it). Every category becomes a
        ``sim_<dictionary>__<category>`` column: the cosine between a text's
        vector and the category's vector, the weighted mean of its terms'
        vectors (cells are weights, ``X`` = 1; wildcards and phrases as LIWC
        reads them). The dictionaries' terms and weights are stored in the
        model, and can be changed later in Settings.
    weighting : {"tokens", "types", "sif"}
        How words are averaged into a text's vector: every occurrence
        (``tokens``), each distinct word once (``types``), or smooth
        inverse frequency (``sif``: the commonest words count least).
    normalize_words : bool
        Scale every word vector to unit length before averaging, so a
        frequent word with a long vector does not dominate.
    probes : str
        Comma-separated words whose nearest neighbors the report shows;
        every concept category's neighbors are shown as well. Empty:
        :data:`DEFAULT_PROBES`, a dozen common content words, skipping any
        the model did not learn. Nothing about training depends on this; it
        only decides which words the clouds are about.
    top_neighbors : int
        Neighbors per probe in the table and the clouds.
    rounding : int
        Decimal places written.

    Returns
    -------
    Path
        ``out_features_csv``.

    Notes
    -----
    Nothing survives below ``min_count`` on a tiny corpus, and the refusal
    says so before gensim is asked. The report beside the model has the
    methods paragraph, the settings, the corpus coverage, the loss per
    epoch, the neighbors, and the package versions -- what a paper needs.
    """
    import numpy as np

    started = time.time()
    missing = _gensim_missing()
    if missing:
        raise ImportError(missing)
    if family not in FAMILIES:
        raise ValueError(f"family must be one of {', '.join(FAMILIES)}, not {family!r}")
    if algorithm not in ALGORITHMS:
        raise ValueError(f"algorithm must be one of {', '.join(ALGORITHMS)}, "
                         f"not {algorithm!r}")
    if weighting not in WEIGHTINGS:
        raise ValueError(f"weighting must be one of {', '.join(WEIGHTINGS)}, "
                         f"not {weighting!r}")
    if int(vector_size) < 2 or int(min_count) < 1 or int(epochs) < 1:
        raise ValueError("vector_size must be at least 2, min_count and epochs "
                         "at least 1")
    dicts = load_concept_dicts(list(concept_dicts))

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    out_features_csv = Path(out_features_csv) if out_features_csv else \
        Path.cwd() / "features" / "word_vectors.csv"
    out_model_json = Path(out_model_json) if out_model_json else \
        out_features_csv.parent / "models" / "word_vectors.json"
    out_neighbors_csv = Path(out_neighbors_csv) if out_neighbors_csv else \
        out_model_json.with_name(out_model_json.stem + "_neighbors.csv")
    out_report_md = Path(out_report_md) if out_report_md else \
        out_model_json.with_name(out_model_json.stem + "_report.md")
    if not overwrite_existing and out_features_csv.is_file() and out_model_json.is_file():
        if verbose:
            print("Word-vector features already exist; returning existing file.")
        return out_features_csv
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)
    out_model_json.parent.mkdir(parents=True, exist_ok=True)

    lowercase = True
    text_settings = {"lemmatize": bool(lemmatize), "pos_tagged": False,
                     "engine": engine, "tokenizer": tokenizer,
                     "stanza_lang": stanza_lang,
                     "keep_punctuation": bool(keep_punctuation),
                     "lowercase": lowercase}

    # 1) first, we tokenize every text once, out to a scratch file that gensim
    #    can stream from. only the type counts stay in memory
    if engine == "stanza":
        announce(on_progress, "loading the stanza pipeline (first use downloads its model)")
    stream = _stream_for(text_settings, device)
    tokens_path = out_model_json.with_name(f".{out_model_json.stem}_tokens.txt")
    ids: List[str] = []
    type_counts: Counter = Counter()
    n_tokens = 0
    from ..helpers.row_map import map_text_rows
    from .ngram_prep import pooled_text_workers

    with atomic_write(tokens_path, mode="w", encoding="utf-8") as fh:
        for row, tokens in map_text_rows(
                analysis_ready, encoding=encoding,
                workers=lambda n_rows: pooled_text_workers(
                    workers, n_rows, engine=engine, tokenizer=tokenizer,
                    lemmatize=bool(lemmatize), pos_tagged=False),
                message="tokenizing texts", on_progress=on_progress,
                inline_fn=lambda pair: _tokens_of(stream, bool(lowercase), pair[1]),
                pool_fn=_tokens_in_worker, initializer=_init_token_worker,
                initargs=(_stream_args(text_settings, device), bool(lowercase))):
            ids.append(str(row.get("text_id", "")))
            clean = [t.replace(" ", "_") for t in tokens if t.strip()]
            type_counts.update(clean)
            n_tokens += len(clean)
            fh.write(" ".join(clean) + "\n")

    surviving = sum(1 for c in type_counts.values() if c >= int(min_count))
    if surviving < 2:
        tokens_path.unlink(missing_ok=True)
        raise ValueError(
            f"only {surviving} word(s) occur at least {min_count} times in "
            f"these {len(ids)} text(s) ({n_tokens} tokens, {len(type_counts)} "
            f"distinct), which is not enough to train on. Lower min_count "
            f"or use a larger corpus.")

    # 2) now we train. gensim streams the scratch file, so what sits in memory
    #    is the vocabulary and the matrix, never the corpus itself
    import gensim

    announce(on_progress, f"training {family} ({algorithm}) for {epochs} epoch(s)")
    threads = 1 if reproducible else max(1, (os.cpu_count() or 2) - 1)
    loss = _EpochLoss()
    progress = _TrainingProgress(on_progress, int(epochs), family)
    common = dict(vector_size=int(vector_size), window=int(window),
                  min_count=int(min_count), sg=1 if algorithm == "skipgram" else 0,
                  negative=int(negative), epochs=int(epochs), seed=int(seed),
                  workers=threads)
    if family == "fasttext":
        # gensim's fastText doesn't keep a running loss, so there's no curve
        # for us to report. the report says so rather than showing zeros
        from gensim.models import FastText

        with progress:
            trained = FastText(corpus_file=str(tokens_path),
                               callbacks=[progress.callback], **common)
    else:
        from gensim.models import Word2Vec

        with progress:
            trained = Word2Vec(corpus_file=str(tokens_path), compute_loss=True,
                               callbacks=[loss.callback, progress.callback], **common)
    kv = trained.wv
    vocabulary = list(kv.index_to_key)
    counts = [int(kv.get_vecattr(w, "count")) for w in vocabulary]
    matrix = np.asarray(kv.vectors, dtype=np.float32)
    losses = [float(x) for x in loss.losses if math.isfinite(x)]
    if not any(losses):
        losses = []          # no loss from fastText; we say so instead of zeros

    # 3) save: the matrix goes beside the manifest, and the manifest names it
    weights = out_model_json.with_suffix(".npy")
    with atomic_write(weights, mode="wb") as fh:
        np.save(fh, matrix)
    covered = sum(c for w, c in type_counts.items() if w in kv.key_to_index)
    oov = [(w, c) for w, c in type_counts.most_common() if w not in kv.key_to_index][:15]
    training = {
        "source": "trained", "family": family, "algorithm": algorithm,
        "vector_size": int(vector_size), "window": int(window),
        "min_count": int(min_count), "epochs": int(epochs),
        "negative": int(negative), "seed": int(seed),
        "reproducible": bool(reproducible), "threads": threads,
        "n_documents": len(ids), "n_tokens": n_tokens,
        "n_types": len(type_counts), "vocabulary_size": len(vocabulary),
        "coverage": (covered / n_tokens) if n_tokens else 0.0,
        "top_oov": oov, "loss_per_epoch": losses,
        "gensim_version": gensim.__version__,
        "wall_seconds": None,     # we fill this in after the report
    }
    model_name = str(name or out_model_json.stem)
    _write_manifest(out_model_json, weights, name=model_name, text=text_settings,
                    vocabulary=vocabulary, counts=counts, dim=int(vector_size),
                    training=training, concept_dicts=dicts,
                    weighting=weighting, normalize_words=normalize_words)
    model = _load_model(out_model_json)

    # 4) features for the training corpus. these go through the very same
    #    row-maker that apply uses, and from the same tokens
    concept_columns, concept_vecs, concept_rows = _concept_vectors(
        model, dicts, stream, bool(lowercase))
    announce(on_progress, "writing the features")
    with atomic_write(out_features_csv, mode="w", newline="", encoding=encoding) as out, \
            tokens_path.open("r", encoding="utf-8") as fh:
        writer = csv.writer(out)
        writer.writerow(_header(model, concept_columns))
        for text_id, line in zip(ids, fh):
            tokens = line.rstrip("\n").split(" ") if line.strip() else []
            writer.writerow([text_id, *_feature_row(
                model, tokens, weighting=weighting,
                normalize_words=normalize_words, concept_vectors=concept_vecs,
                rounding=rounding)])
    tokens_path.unlink(missing_ok=True)

    # 5) lastly, the neighbors (of the probe words, and of every concept
    #    category) and the report
    probe_words = _probe_words(probes, vocabulary, counts)
    neighbors = _neighbors(model, probe_words, stream, top_n=int(top_neighbors))
    neighbors.update(_neighbors_of_vectors(model, concept_columns, concept_vecs,
                                             top_n=int(top_neighbors)))
    _write_neighbors(out_neighbors_csv, neighbors, encoding=encoding,
                      rounding=rounding)
    training["wall_seconds"] = round(time.time() - started, 1)
    doc = json.loads(out_model_json.read_text(encoding="utf-8"))
    doc["training"] = training
    with atomic_write(out_model_json, mode="w", encoding="utf-8") as fh:
        json.dump(doc, fh, indent=1)
    _write_report(out_report_md, doc, out_model_json, weights, neighbors,
                  concept_rows, features_csv=out_features_csv,
                  neighbors_csv=out_neighbors_csv, model=model)
    if verbose:
        print(f"[word_vectors] {len(vocabulary)} words x {vector_size} dims "
              f"from {len(ids)} text(s); coverage "
              f"{100 * training['coverage']:.1f}% -> {out_model_json}")
        for row in concept_rows:
            if row["missed"]:
                print(f"[word_vectors] note: {row['column']}: {len(row['missed'])} of "
                      f"{row['n_terms']} term(s) not in the vocabulary")
    return out_features_csv

taters.text.subtitle_parser

SubtitleSegment dataclass

SubtitleSegment(number, start_ms, end_ms, text, name=None)

Normalized subtitle cue spanning a time interval.

Parameters:

Name	Type	Description	Default
number	int or None	SRT block index if present; None for VTT or SRTs without explicit numbering.	required
start_ms	int	Start time in milliseconds.	required
end_ms	int	End time in milliseconds.	required
text	str	Cue text content. May contain embedded newlines if the source had multiple lines.	required
name	str or None	Optional speaker/name field (not populated by the built-in parsers).	None

Notes

Instances are immutable (frozen=True) so they can be safely shared and hashed.

convert_subtitles

convert_subtitles(
    *,
    input,
    to,
    output=None,
    encoding=None,
    include_name=False,
    overwrite_existing=False
)

Convert an SRT/VTT file to CSV/SRT/VTT.

Reads a subtitle file, parses into normalized segments, and renders to the requested format. When output is omitted, a default path is created at ./features/subtitles/<input_stem>.<ext>.

Parameters:

Name	Type	Description	Default
input	str or Path	Path to the input .srt or .vtt file.	required
to	('csv', 'srt', 'vtt')	Desired output format.	'csv'
output	str or Path	Explicit output path. If None, use the default location.	None
encoding	str	Input encoding override; otherwise auto-detected (or UTF-8).	None
include_name	bool	When to='csv', include a name column if available.	False
overwrite_existing	bool	If False and the output already exists, return it untouched rather than re-rendering, matching the rest of Taters.	False

Returns:

Type	Description
Path	Path to the written output file.

Raises:

Type	Description
FileNotFoundError	If the input file does not exist.
ValueError	If the output format is unsupported or input content is malformed.

Source code in src\taters\text\subtitle_parser.py

def convert_subtitles(
    *,
    input: Union[str, Path],
    to: Literal["csv", "srt", "vtt"],
    output: Optional[Union[str, Path]] = None,
    encoding: Optional[str] = None,
    include_name: bool = False,
    overwrite_existing: bool = False,
) -> Path:
    """
    Convert an SRT/VTT file to CSV/SRT/VTT.

    Reads a subtitle file, parses into normalized segments, and renders to the
    requested format. When ``output`` is omitted, a default path is created at
    ``./features/subtitles/<input_stem>.<ext>``.

    Parameters
    ----------
    input : str or pathlib.Path
        Path to the input ``.srt`` or ``.vtt`` file.
    to : {'csv', 'srt', 'vtt'}
        Desired output format.
    output : str or pathlib.Path, optional
        Explicit output path. If ``None``, use the default location.
    encoding : str, optional
        Input encoding override; otherwise auto-detected (or UTF-8).
    include_name : bool, default=False
        When ``to='csv'``, include a ``name`` column if available.
    overwrite_existing : bool, default=False
        If ``False`` and the output already exists, return it untouched rather
        than re-rendering, matching the rest of Taters.

    Returns
    -------
    pathlib.Path
        Path to the written output file.

    Raises
    ------
    FileNotFoundError
        If the input file does not exist.
    ValueError
        If the output format is unsupported or input content is malformed.
    """

    ext_for = {"csv": ".csv", "srt": ".srt", "vtt": ".vtt"}
    if to not in ext_for:
        # we check this up front. otherwise, an unknown format with an explicit
        # `output` path would fall straight through to the VTT renderer
        raise ValueError(f"Unsupported output format: {to!r}. Choose from csv, srt, vtt.")

    in_path = Path(input).resolve()
    if not in_path.exists():
        raise FileNotFoundError(f"Subtitle file not found: {in_path}")

    # if we weren't given an output path, we make one up
    if output is not None:
        out_path = Path(output)
    else:
        out_path = _default_out_dir() / f"{in_path.stem}{ext_for[to]}"

    if out_path.exists() and not overwrite_existing:
        print(f"Subtitle output already exists; returning existing file: {out_path}")
        return out_path

    segs = parse_subtitles(in_path, encoding=encoding)

    # now we render
    if to == "csv":
        return render_to_csv(segs, out_path, include_name=include_name)
    elif to == "srt":
        return render_to_srt(segs, out_path)
    else:  # "vtt"
        return render_to_vtt(segs, out_path)

main

main()

Command-line entry point for subtitle parsing and conversion.

Parses arguments via :func:_build_arg_parser, calls :func:convert_subtitles, and prints the resulting output path.

Examples:

$ python -m taters.text.subtitle_parser --input transcript.srt --to csv --output features/subtitles/transcript.csv

Source code in src\taters\text\subtitle_parser.py

def main():
    """
    Command-line entry point for subtitle parsing and conversion.

    Parses arguments via :func:`_build_arg_parser`, calls
    :func:`convert_subtitles`, and prints the resulting output path.

    Examples
    --------
    $ python -m taters.text.subtitle_parser \
        --input transcript.srt --to csv \
        --output features/subtitles/transcript.csv
    """

    args = _build_arg_parser().parse_args()
    out = convert_subtitles(
        input=args.input,
        to=args.to,
        output=args.output,
        encoding=args.encoding,
        include_name=args.include_name,
        overwrite_existing=args.overwrite_existing,
    )
    print(str(out))

parse_srt

parse_srt(text)

Parse SRT content into normalized subtitle segments.

The parser tolerates extra whitespace and the optional numeric index line. Each cue must include a timestamp line of the form HH:MM:SS,mmm --> HH:MM:SS,mmm (a dot separator for milliseconds is also accepted for robustness).

Parameters:

Name	Type	Description	Default
text	str	Entire SRT file content.	required

Returns:

Type	Description
list[SubtitleSegment]	Parsed cues with millisecond times and original (joined) text.

Raises:

Type	Description
ValueError	If a well-formed timestamp line is missing where expected.

Source code in src\taters\text\subtitle_parser.py

def parse_srt(text: str) -> List[SubtitleSegment]:
    """
    Parse SRT content into normalized subtitle segments.

    The parser tolerates extra whitespace and the optional numeric index line.
    Each cue must include a timestamp line of the form
    ``HH:MM:SS,mmm --> HH:MM:SS,mmm`` (a dot separator for milliseconds is also
    accepted for robustness).

    Parameters
    ----------
    text : str
        Entire SRT file content.

    Returns
    -------
    list[SubtitleSegment]
        Parsed cues with millisecond times and original (joined) text.

    Raises
    ------
    ValueError
        If a well-formed timestamp line is missing where expected.
    """

    lines = [ln.rstrip("\r") for ln in text.splitlines()]
    i = 0
    n = len(lines)
    out: List[SubtitleSegment] = []

    while i < n:
        # first, skip past any blank lines
        while i < n and lines[i].strip() == "":
            i += 1
        if i >= n:
            break

        # the numeric index, if this block has one
        number = None
        maybe_num = lines[i].strip()
        ts_line_idx = i
        if maybe_num.isdigit():
            number = int(maybe_num)
            i += 1
            ts_line_idx = i

        if i >= n:
            break

        # now the timestamp line
        m = _SRT_TS_LINE.match(lines[ts_line_idx].strip())
        if not m:
            # some SRTs skip the numeric index entirely, so we let the
            # timestamp come first
            m = _SRT_TS_LINE.match(lines[i].strip())
            if not m:
                raise ValueError(f"SRT parse error: expected timestamp near line {ts_line_idx+1}")
            ts_line_idx = i
        i = ts_line_idx + 1

        start_ms = _parse_timestamp(m.group("start"))
        end_ms = _parse_timestamp(m.group("end"))

        # then the content lines, up until the next blank
        content: List[str] = []
        while i < n and lines[i].strip() != "":
            content.append(lines[i])
            i += 1

        if not content:
            # SRT allows empty cues, so we keep things consistent and treat
            # one as an empty string
            content = [""]

        text_block = "\n".join(content)
        out.append(SubtitleSegment(number=number, start_ms=start_ms, end_ms=end_ms, text=text_block, name=None))

        # lastly, skip the trailing blank between blocks
        while i < n and lines[i].strip() == "":
            i += 1

    return out

parse_subtitles

parse_subtitles(input_path, *, encoding=None)

Auto-detect and parse a subtitle file by extension.

.vtt files are parsed as WebVTT; .srt and unknown extensions are parsed as SRT. Input encoding is detected with chardet when available, otherwise UTF-8 is assumed. Decoding errors are replaced.

Parameters:

Name	Type	Description	Default
input_path	str or Path	Path to an SRT or VTT file.	required
encoding	str	Override input encoding. If omitted, try detect then fall back to UTF-8.	None

Returns:

Type	Description
list[SubtitleSegment]	Normalized subtitle segments.

Raises:

Type	Description
FileNotFoundError	If the path does not exist.

Source code in src\taters\text\subtitle_parser.py

def parse_subtitles(input_path: Union[str, Path], *, encoding: Optional[str] = None) -> List[SubtitleSegment]:
    """
    Auto-detect and parse a subtitle file by extension.

    ``.vtt`` files are parsed as WebVTT; ``.srt`` and unknown extensions are
    parsed as SRT. Input encoding is detected with ``chardet`` when available,
    otherwise UTF-8 is assumed. Decoding errors are replaced.

    Parameters
    ----------
    input_path : str or pathlib.Path
        Path to an SRT or VTT file.
    encoding : str, optional
        Override input encoding. If omitted, try detect then fall back to UTF-8.

    Returns
    -------
    list[SubtitleSegment]
        Normalized subtitle segments.

    Raises
    ------
    FileNotFoundError
        If the path does not exist.
    """

    path = Path(input_path)
    if not path.exists():
        raise FileNotFoundError(f"Subtitle file not found: {path}")

    enc = encoding or _detect_encoding(path)
    raw = path.read_text(encoding=enc, errors="replace")

    ext = path.suffix.lower()
    if ext == ".vtt":
        return parse_vtt(raw)
    else:
        # anything that isn't .vtt (including unknown extensions, which are
        # common in the wild) gets treated as SRT
        return parse_srt(raw)

parse_vtt

parse_vtt(text)

Parse WebVTT content into normalized subtitle segments.

Behavior: - Skips the WEBVTT header and any header metadata. - Skips NOTE and STYLE blocks. - Ignores optional cue identifiers. - Requires a timestamp line of the form HH:MM:SS.mmm --> HH:MM:SS.mmm (comma also accepted).

Parameters:

Name	Type	Description	Default
text	str	Entire VTT file content.	required

Returns:

Type	Description
list[SubtitleSegment]	Parsed cues with millisecond times and original (joined) text.

Raises:

Type	Description
ValueError	If a required timestamp line is malformed or missing.

Source code in src\taters\text\subtitle_parser.py

def parse_vtt(text: str) -> List[SubtitleSegment]:
    """
    Parse WebVTT content into normalized subtitle segments.

    Behavior:
    - Skips the ``WEBVTT`` header and any header metadata.
    - Skips ``NOTE`` and ``STYLE`` blocks.
    - Ignores optional cue identifiers.
    - Requires a timestamp line of the form
    ``HH:MM:SS.mmm --> HH:MM:SS.mmm`` (comma also accepted).

    Parameters
    ----------
    text : str
        Entire VTT file content.

    Returns
    -------
    list[SubtitleSegment]
        Parsed cues with millisecond times and original (joined) text.

    Raises
    ------
    ValueError
        If a required timestamp line is malformed or missing.
    """

    lines = [ln.rstrip("\r") for ln in text.splitlines()]
    i = 0
    n = len(lines)

    # the header
    if i < n and lines[i].strip().upper().startswith("WEBVTT"):
        i += 1
        # skip the header metadata, up to the blank line
        while i < n and lines[i].strip() != "":
            i += 1
        while i < n and lines[i].strip() == "":
            i += 1

    out: List[SubtitleSegment] = []

    while i < n:
        # NOTE and STYLE blocks aren't cues, so we skip them
        if lines[i].strip().startswith("NOTE") or lines[i].strip().upper() == "STYLE":
            # ...all the way to the blank line
            i += 1
            while i < n and lines[i].strip() != "":
                i += 1
            while i < n and lines[i].strip() == "":
                i += 1
            continue

        # cues can have an identifier line before the timestamp. we don't use
        # it, so: if this line has '-->' it's the timestamp, otherwise we peek
        # at the next line (two lines of lookahead, max)
        if i < n and "-->" not in lines[i]:
            # might be an identifier; check the next line
            if i + 1 < n and "-->" in lines[i + 1]:
                i += 1  # eat the ID and ignore its value
            # otherwise we fall through, and the timestamp check below complains

        if i >= n:
            break

        # the timestamp line
        line = lines[i].strip()
        if "-->" not in line:
            raise ValueError(f"VTT parse error: expected timestamp at line {i+1}")
        parts = [p.strip() for p in line.split("-->")]
        if len(parts) < 2:
            raise ValueError(f"VTT parse error: invalid timestamp at line {i+1}")

        start_ms = _parse_timestamp(parts[0])
        end_ms = _parse_timestamp(parts[1].split(" ")[0])  # drop any cue settings
        i += 1

        # the content, up until the next blank
        content: List[str] = []
        while i < n and lines[i].strip() != "":
            content.append(lines[i])
            i += 1

        if not content:
            content = [""]

        text_block = "\n".join(content)
        out.append(SubtitleSegment(number=None, start_ms=start_ms, end_ms=end_ms, text=text_block, name=None))

        while i < n and lines[i].strip() == "":
            i += 1

    return out

render_to_csv

render_to_csv(segs, out_path, *, include_name=False)

Write segments to a CSV file.

The CSV schema is:

start_time,end_time[,name],text

Times are written as integer milliseconds (stringified) to preserve exact alignment for downstream tools.

Parameters:

Name	Type	Description	Default
segs	Iterable[SubtitleSegment]	Segments to write.	required
out_path	str or Path	Output CSV path.	required
include_name	bool	Include a name column (useful if upstream added speaker names).	False

Returns:

Type	Description
Path	Path to the written CSV file.

Source code in src\taters\text\subtitle_parser.py

def render_to_csv(segs: Iterable[SubtitleSegment], out_path: Union[str, Path], *, include_name: bool = False) -> Path:
    """
    Write segments to a CSV file.

    The CSV schema is:

    ``start_time,end_time[,name],text``

    Times are written as integer milliseconds (stringified) to preserve exact
    alignment for downstream tools.

    Parameters
    ----------
    segs : Iterable[SubtitleSegment]
        Segments to write.
    out_path : str or pathlib.Path
        Output CSV path.
    include_name : bool, default=False
        Include a ``name`` column (useful if upstream added speaker names).

    Returns
    -------
    pathlib.Path
        Path to the written CSV file.
    """

    out_path = Path(out_path)
    out_path.parent.mkdir(parents=True, exist_ok=True)
    with atomic_write(out_path, mode="w", encoding="utf-8", newline="") as f:
        w = csv.writer(f)
        header = ["start_time", "end_time"]
        if include_name:
            header.append("name")
        header.append("text")
        w.writerow(header)
        for s in segs:
            row: List[str] = [f"{s.start_ms}", f"{s.end_ms}"]
            if include_name:
                row.append(s.name or "")
            row.append(s.text)
            w.writerow(row)
    return out_path

render_to_srt

render_to_srt(segs, out_path)

Write segments to SRT format.

Blocks are 1-indexed and use HH:MM:SS,mmm timestamps.

Parameters:

Name	Type	Description	Default
segs	Iterable[SubtitleSegment]	Segments to write.	required
out_path	str or Path	Output .srt path.	required

Returns:

Type	Description
Path	Path to the written SRT file.

Source code in src\taters\text\subtitle_parser.py

def render_to_srt(segs: Iterable[SubtitleSegment], out_path: Union[str, Path]) -> Path:
    """
    Write segments to SRT format.

    Blocks are 1-indexed and use ``HH:MM:SS,mmm`` timestamps.

    Parameters
    ----------
    segs : Iterable[SubtitleSegment]
        Segments to write.
    out_path : str or pathlib.Path
        Output ``.srt`` path.

    Returns
    -------
    pathlib.Path
        Path to the written SRT file.
    """

    out_path = Path(out_path)
    out_path.parent.mkdir(parents=True, exist_ok=True)
    with atomic_write(out_path, mode="w", encoding="utf-8", newline="") as f:
        for i, s in enumerate(segs, start=1):
            f.write(f"{i}\n")
            f.write(f"{_fmt_ms_srt(s.start_ms)} --> {_fmt_ms_srt(s.end_ms)}\n")
            f.write(f"{s.text}\n\n")
    return out_path

render_to_vtt

render_to_vtt(segs, out_path)

Write segments to WebVTT format.

Includes a standard WEBVTT header and uses HH:MM:SS.mmm timestamps.

Parameters:

Name	Type	Description	Default
segs	Iterable[SubtitleSegment]	Segments to write.	required
out_path	str or Path	Output .vtt path.	required

Returns:

Type	Description
Path	Path to the written VTT file.

Source code in src\taters\text\subtitle_parser.py

def render_to_vtt(segs: Iterable[SubtitleSegment], out_path: Union[str, Path]) -> Path:
    """
    Write segments to WebVTT format.

    Includes a standard ``WEBVTT`` header and uses ``HH:MM:SS.mmm`` timestamps.

    Parameters
    ----------
    segs : Iterable[SubtitleSegment]
        Segments to write.
    out_path : str or pathlib.Path
        Output ``.vtt`` path.

    Returns
    -------
    pathlib.Path
        Path to the written VTT file.
    """

    out_path = Path(out_path)
    out_path.parent.mkdir(parents=True, exist_ok=True)
    with atomic_write(out_path, mode="w", encoding="utf-8", newline="") as f:
        f.write("WEBVTT\n\n")
        for s in segs:
            f.write(f"{_fmt_ms_vtt(s.start_ms)} --> {_fmt_ms_vtt(s.end_ms)}\n")
            f.write(f"{s.text}\n\n")
    return out_path

taters.text.analyze_ngram_frequencies

Corpus-level n-gram frequency list, with collocation statistics.

One output row per retained n-gram (all orders 1..ngram_n), not one row per text: this is a corpus tool. "Document" means one analysis-ready row, so the same unit-of-analysis machinery the other text steps use decides what NPMI's document counts mean here too.

The statistics follow Bouma (2009), "Normalized (Pointwise) Mutual Information in Collocation Extraction", with the chain split NPMI((w1..wn-1), wn) for orders above two. Three corrections over the C# plugin this replaces (BUTTER's Frequency List), all silent-result bugs rather than crashes:

• Every probability shares one sample space: the corpus token count -- p(g) = freq(g) / N for the n-gram and both of its parts, which is the formulation Bouma's bounds are proved for (and what e.g. gensim's phrase scorer computes). The plugin normalized each order by its own total (bigrams by the bigram count, words by the word count), mixing sample spaces -- under which NPMI is not bounded by 1 and "0.9" means nothing.
• Those counts are the full counts, taken before any minimum-frequency or document-share filter removes rows. Computing totals after filtering (as the plugin did) inflates every probability by exactly what was filtered, so NPMI values drifted with the user's filter settings.
• Nothing is pruned during counting, so every retained n-gram's subgram counts exist and every retained n-gram gets a score. The plugin pruned periodically for memory and then silently discarded any n-gram whose subgram had been pruned out from under it. Memory is bounded a different way: past a RAM budget (max_ram_mb) the counting table spills sorted batches to disk and merges them once at the end (helpers.spill_counter) -- the counts stay exact and the output file stays identical.

Filters (minimum frequency, minimum document share, the stoplist, and the optional NPMI/logDice thresholds) apply only at write time, to output rows -- never to the counts the statistics are computed from.

analyze_ngram_frequencies

analyze_ngram_frequencies(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    on_progress=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    ngram_n=1,
    lemmatize=False,
    pos_tagged=False,
    engine="nltk",
    tokenizer="potts",
    stanza_lang="en",
    keep_punctuation=False,
    device="auto",
    stoplist_paths=None,
    min_freq=5,
    min_obs_pct=0.1,
    min_token_count=10,
    min_npmi=None,
    min_logdice=None,
    max_ram_mb=1024,
    rounding=4
)

Build a corpus frequency list of 1..ngram_n-grams and write it as CSV.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of .txt files, or a prebuilt analysis-ready CSV.	None
txt_dir	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of .txt files, or a prebuilt analysis-ready CSV.	None
analysis_csv	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of .txt files, or a prebuilt analysis-ready CSV.	None
gathered_csv	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of .txt files, or a prebuilt analysis-ready CSV.	None
out_features_csv	str or Path	Output file path. If None, defaults to ./features/ngram_frequencies/<analysis_ready_filename>.	None
overwrite_existing	bool	If False and the output file already exists, skip processing and return the path. Also controls whether the intermediate analysis-ready CSV is rebuilt from the current source or reused.	False
encoding	str	Encoding for reading and writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	When gathering from a CSV, name(s) of the column(s) containing text.	("text",)
id_cols	Sequence[str] or None	Optional ID columns that identify each row when gathering from CSV.	None
mode	('concat', 'separate')	Gathering behavior when multiple text columns are provided: "concat" joins them into one text per row; "separate" measures each column on its own.	"concat"
group_by	Sequence[str] or None	Optional grouping keys used during CSV gathering (e.g. ["speaker"]) -- one document per group instead of one per row.	None
delimiter	str	Column separator of the input CSV.	","
pattern	str	Which files to read when gathering from a folder of documents (globs, ;-separated). Only used with txt_dir.	every document type
ngram_n	int	Highest n-gram order. The output lists every order from 1 up to this, and the lower orders are needed internally for NPMI in any case.	1
lemmatize	bool	Lemmatize tokens (WordNet, POS-guided) before counting. Always applied before the stoplist, so "be" catches "is/was/were".	False
pos_tagged	bool	Treat word+tag as the unit: the verb "felt" and the noun "felt" become separate rows, and a pos column (after ngram) names each term's tag(s). Off, the column does not exist at all.	False
stoplist_paths	sequence of paths	Stop word/character lists (.txt, one entry per line; folders are expanded). None means a vanilla list. Applied to finished n-grams -- a row is dropped when any of its tokens is a stop entry -- never to the token stream, so counts and NPMI keep their true values.	None
engine	('nltk', 'stanza')	Who tags: NLTK's perceptron tagger, or Stanza's neural pipeline. Stanza is slower, more accurate, multilingual, and GPU-optional.	"nltk"
tokenizer	('potts', 'stanza')	Who splits text into tokens. The default (the Potts social-media tokenizer) keeps counts comparable across engines and keeps emoticons, hashtags and URLs whole; "stanza" (Stanza engine only) hands Stanza the whole job.	"potts"
stanza_lang	str	Language for the Stanza engine; that language's model is downloaded once on first use (can be a few hundred MB). Only with engine="stanza".	"en"
keep_punctuation	bool	Count punctuation (and emoticons) as terms. Off, only tokens with a letter or digit are counted, so "." and "," never become vocabulary. Must match across the frequency list, the matrix and the topic model.	False
device	('auto', 'cuda', 'cpu')	Where Stanza runs: "auto", "cuda", or "cpu". Only with engine="stanza"; the NLTK engine is CPU-only either way.	"auto"
min_freq	int	Drop n-grams rarer than this from the output.	5
min_obs_pct	float	Drop n-grams appearing in fewer than this percent of documents.	0.10
min_token_count	int	Skip documents shorter than this many tokens entirely (they do not count toward document totals either).	10
min_npmi	float	Optional collocation thresholds for orders above one. Default None: the metrics are reported and filtering stays an analysis decision.	None
min_logdice	float	Optional collocation thresholds for orders above one. Default None: the metrics are reported and filtering stays an analysis decision.	None
max_ram_mb	int	Approximate RAM budget (in MB) for the n-gram counting table. A corpus whose vocabulary outgrows it has sorted batches cached to a temporary folder and merged once at the end: the output file is identical, memory stays bounded, and the disk is touched in a few big sequential passes rather than per gram. A small corpus never reaches the budget and never notices; raise it on a big machine for speed on a huge corpus.	1024
workers	int	Parallel processes, spent twice: reading documents during the gather, then tokenizing-and-tallying during the count. 0 means automatic: three-quarters of the logical cores; 1 turns parallelism off. Output files are identical whatever the worker count. Counting under a Stanza model stays single-process (its lever is batching, not processes). Each in-flight document holds its gram tallies in memory, so many workers over very large documents costs RAM beyond max_ram_mb's table.	0
rounding	int	Decimal places for the derived statistics.	4

Returns:

Type	Description
Path	out_features_csv, sorted by descending frequency (ties broken alphabetically), rank assigned after all filters.

Raises:

Type	Description
ValueError	If no document met min_token_count -- an empty frequency list is indistinguishable from a misconfigured run, so it refuses instead.

Source code in src\taters\text\analyze_ngram_frequencies.py

@records_settings(binding=TEXT_INPUT, grain=TEXT_GRAIN,
                  outputs=("out_features_csv",),
                  assets={"stoplist_paths": "stoplists"})
def analyze_ngram_frequencies(
    *,
    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    on_progress: Optional[Callable[[int, int], None]] = None,

    # ----- Output -----
    out_features_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    workers: int = 0,

    # ====== SHARED I/O OPTIONS ======
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS (used when csv_path is provided) ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ====== TXT FOLDER GATHER OPTIONS (used when txt_dir is provided) ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ====== N-GRAM OPTIONS ======
    ngram_n: int = 1,
    lemmatize: bool = False,
    pos_tagged: bool = False,
    engine: Literal["nltk", "stanza"] = "nltk",
    tokenizer: Literal["potts", "stanza"] = "potts",
    stanza_lang: str = "en",
    keep_punctuation: bool = False,
    device: str = "auto",
    stoplist_paths: Optional[Sequence[PathLike]] = None,
    min_freq: int = 5,
    min_obs_pct: float = 0.10,
    min_token_count: int = 10,
    min_npmi: Optional[float] = None,
    min_logdice: Optional[float] = None,
    max_ram_mb: int = 1024,
    rounding: int = 4,
) -> Path:
    """
    Build a corpus frequency list of 1..``ngram_n``-grams and write it as CSV.

    Parameters
    ----------
    csv_path, txt_dir, analysis_csv, gathered_csv
        The same input contract as the other text analyzers: a spreadsheet of
        texts, a folder of ``.txt`` files, or a prebuilt analysis-ready CSV.
    out_features_csv : str or pathlib.Path, optional
        Output file path. If ``None``, defaults to
        ``./features/ngram_frequencies/<analysis_ready_filename>``.
    overwrite_existing : bool, default=False
        If ``False`` and the output file already exists, skip processing and
        return the path. Also controls whether the intermediate analysis-ready
        CSV is rebuilt from the current source or reused.
    encoding : str, default="utf-8-sig"
        Encoding for reading and writing CSV files.
    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV, name(s) of the column(s) containing text.
    id_cols : Sequence[str] or None, optional
        Optional ID columns that identify each row when gathering from CSV.
    mode : {"concat", "separate"}, default="concat"
        Gathering behavior when multiple text columns are provided:
        ``"concat"`` joins them into one text per row; ``"separate"`` measures
        each column on its own.
    group_by : Sequence[str] or None, optional
        Optional grouping keys used during CSV gathering (e.g. ``["speaker"]``)
        -- one document per group instead of one per row.
    delimiter : str, default=","
        Column separator of the *input* CSV.
    pattern : str, default=every document type
        Which files to read when gathering from a folder of documents
        (globs, ``;``-separated). Only used with ``txt_dir``.
    ngram_n : int, default=1
        Highest n-gram order. The output lists every order from 1 up to this,
        and the lower orders are needed internally for NPMI in any case.
    lemmatize : bool, default=False
        Lemmatize tokens (WordNet, POS-guided) before counting. Always
        applied *before* the stoplist, so "be" catches "is/was/were".
    pos_tagged : bool, default=False
        Treat word+tag as the unit: the verb "felt" and the noun "felt"
        become separate rows, and a ``pos`` column (after ``ngram``) names
        each term's tag(s). Off, the column does not exist at all.
    stoplist_paths : sequence of paths, optional
        Stop word/character lists (.txt, one entry per line; folders are
        expanded). None means a vanilla list. Applied to finished n-grams --
        a row is dropped when any of its tokens is a stop entry -- never to
        the token stream, so counts and NPMI keep their true values.
    engine : {"nltk", "stanza"}, default="nltk"
        Who tags: NLTK's perceptron tagger, or Stanza's neural pipeline.
        Stanza is slower, more accurate, multilingual, and GPU-optional.
    tokenizer : {"potts", "stanza"}, default="potts"
        Who splits text into tokens. The default (the Potts social-media
        tokenizer) keeps counts comparable across engines and keeps
        emoticons, hashtags and URLs whole; "stanza" (Stanza engine only)
        hands Stanza the whole job.
    stanza_lang : str, default="en"
        Language for the Stanza engine; that language's model is downloaded
        once on first use (can be a few hundred MB). Only with engine="stanza".
    keep_punctuation : bool, default=False
        Count punctuation (and emoticons) as terms. Off, only tokens with a
        letter or digit are counted, so "." and "," never become vocabulary.
        Must match across the frequency list, the matrix and the topic model.
    device : {"auto", "cuda", "cpu"}, default="auto"
        Where Stanza runs: "auto", "cuda", or "cpu". Only with engine="stanza";
        the NLTK engine is CPU-only either way.
    min_freq : int, default=5
        Drop n-grams rarer than this from the output.
    min_obs_pct : float, default=0.10
        Drop n-grams appearing in fewer than this *percent* of documents.
    min_token_count : int, default=10
        Skip documents shorter than this many tokens entirely (they do not
        count toward document totals either).
    min_npmi, min_logdice : float, optional
        Optional collocation thresholds for orders above one. Default None:
        the metrics are reported and filtering stays an analysis decision.
    max_ram_mb : int, default=1024
        Approximate RAM budget (in MB) for the n-gram counting table. A
        corpus whose vocabulary outgrows it has sorted batches cached to a
        temporary folder and merged once at the end: the output file is
        identical, memory stays bounded, and the disk is touched in a few
        big sequential passes rather than per gram. A small corpus never
        reaches the budget and never notices; raise it on a big machine for
        speed on a huge corpus.
    workers : int, default=0
        Parallel processes, spent twice: reading documents during the gather,
        then tokenizing-and-tallying during the count. ``0`` means automatic:
        three-quarters of the logical cores; ``1`` turns parallelism off.
        Output files are identical whatever the worker count. Counting under
        a Stanza model stays single-process (its lever is batching, not
        processes). Each in-flight document holds its gram tallies in memory,
        so many workers over very large documents costs RAM beyond
        ``max_ram_mb``'s table.
    rounding : int, default=4
        Decimal places for the derived statistics.

    Returns
    -------
    Path
        ``out_features_csv``, sorted by descending frequency (ties broken
        alphabetically), rank assigned after all filters.

    Raises
    ------
    ValueError
        If no document met ``min_token_count`` -- an empty frequency list is
        indistinguishable from a misconfigured run, so it refuses instead.
    """
    if ngram_n < 1:
        raise ValueError(f"ngram_n must be >= 1, got {ngram_n}")

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        out_features_csv = Path.cwd() / "features" / "ngram_frequencies" / analysis_ready.name
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)

    if not overwrite_existing and out_features_csv.is_file():
        print("N-gram frequency output file already exists; returning existing file.")
        return out_features_csv

    if engine == "stanza" and (lemmatize or pos_tagged or tokenizer == "stanza"):
        # building the stanza pipeline might download its model, so we name
        # the phase. otherwise the display just sits on the previous one
        # while that happens
        announce(on_progress, "loading the stanza pipeline (first use "
                              "downloads its model)")
    stream = make_token_stream(lemmatize, pos_tagged, engine=engine,
                               keep_punctuation=keep_punctuation,
                               tokenizer=tokenizer,
                               stanza_lang=stanza_lang, device=device)
    stopset = load_stoplist(stoplist_paths) if stoplist_paths else set()

    # 2) one pass over everything: we count every n-gram of every order, plus
    #    the corpus token count that every probability hangs off of. no pruning
    #    here, whatever the filters say -- see the module docstring for why.
    #    the counter keeps the table in RAM up to `max_ram_mb` and spills
    #    sorted batches to disk past that; either way, the counts are exact
    counter = SpillCounter(
        max(0, int(max_ram_mb)) * 1024 * 1024, tmp_root=tmp_root,
        on_spill=lambda batch: announce(
            on_progress, f"cached n-gram counts to disk (batch {batch})"),
    )
    total_tokens = 0
    n_docs = 0

    try:
        # tokenizing is the expensive (and embarrassingly parallel) half, so
        # that and the per-document tallying go to the pool. folding into the
        # shared counter stays here, in submission order -- that's what keeps
        # the output identical no matter the worker count, spills included
        total_rows = count_rows(analysis_ready, on_progress=on_progress)
        count_workers = _counting_workers(workers, total_rows, engine=engine,
                                          tokenizer=tokenizer,
                                          lemmatize=lemmatize,
                                          pos_tagged=pos_tagged)
        stream_args = dict(lemmatize=lemmatize, pos_tagged=pos_tagged,
                           engine=engine, tokenizer=tokenizer,
                           stanza_lang=stanza_lang, device=device,
                           keep_punctuation=keep_punctuation)
        reporter = FlightReporter(on_progress, total_rows, "counting n-grams")
        with analysis_ready.open("r", newline="", encoding=encoding) as f:
            reader = csv.DictReader(f)
            fields = reader.fieldnames or []
            if "text_id" not in fields or "text" not in fields:
                raise ValueError(
                    f"Expected columns 'text_id' and 'text' in {analysis_ready}; found {fields}"
                )

            def pairs():
                for row in reader:
                    yield (str(row.get("text_id", "") or ""),
                           str(row.get("text", "") or ""))

            for result in ordered_parallel_map(
                    lambda pair: _doc_grams(stream, pair[1], ngram_n,
                                            min_token_count),
                    pairs(),
                    workers=count_workers,
                    pool_fn=_grams_in_worker,
                    initializer=_init_gram_worker,
                    initargs=(stream_args, ngram_n, min_token_count),
                    on_start=lambda pair: reporter.start(pair[0]),
                    on_finish=lambda pair: reporter.finish(pair[0])):
                reporter.consumed()
                if result is None:
                    continue
                n_tokens, doc_counts = result
                n_docs += 1
                total_tokens += n_tokens
                counter.add_doc(doc_counts)

        if n_docs == 0:
            raise ValueError(
                f"No document reached min_token_count={min_token_count}; "
                "an empty frequency list would look like a run that worked."
            )

        # 3) score and filter. the statistics always come from the full counts
        #    above; the filters only pick which rows make it to the file. both
        #    paths share the same arithmetic (`_collocation_stats`) and give
        #    the same rows -- the disk path just looks up subgram counts
        #    without holding the full table in memory
        opts = _FilterOpts(n_docs=n_docs, total_tokens=total_tokens,
                           min_freq=min_freq, min_obs_pct=min_obs_pct,
                           stopset=stopset, min_npmi=min_npmi,
                           min_logdice=min_logdice)
        if counter.spilled:
            announce(on_progress, "merging n-gram counts cached on disk")
            kept = _score_streaming(counter, opts)
        else:
            kept = _score_in_memory(counter.data, opts)
    finally:
        counter.close()

    # ties break on the words (then the tags), so that "felt (NN)" and
    # "felt (VBD)" at the same frequency land in the same order every run
    kept.sort(key=lambda row: (-row[2], words_of(row[0]), tags_of(row[0])))

    with atomic_write(out_features_csv, newline="", encoding=encoding) as f:
        writer = csv.writer(f)
        writer.writerow(header_for(pos_tagged))
        for rank, (gram, n, freq, docs, obs_pct, npmi, logdice) in enumerate(kept, 1):
            row = [
                rank, words_of(gram), n, freq, docs,
                round(obs_pct, rounding),
                round(math.log(n_docs / docs), rounding),
                "" if npmi is None else round(npmi, rounding),
                "" if logdice is None else round(logdice, rounding),
            ]
            if pos_tagged:
                row.insert(2, tags_of(gram))
            writer.writerow(row)

    return out_features_csv

header_for

header_for(pos_tagged)

The header this run will write; pos exists only when asked for.

Source code in src\taters\text\analyze_ngram_frequencies.py

def header_for(pos_tagged: bool) -> List[str]:
    """The header this run will write; ``pos`` exists only when asked for."""
    if not pos_tagged:
        return list(HEADER)
    return HEADER[:2] + ["pos"] + HEADER[2:]

taters.text.build_doc_term_matrix

Document-term matrix built over a vocabulary taken from a frequency list.

The companion step to :mod:analyze_ngram_frequencies, and the on-ramp for the topic-modeling steps planned behind it: one row per document, one column per retained term, cells weighted as counts, binary, relative frequency, or TF-IDF. "Document" means one analysis-ready row, exactly as in the frequency list, so the two steps agree about units by construction.

Two agreements with the frequency list are load-bearing:

• The token stream must match. The vocabulary was built from prepared (possibly lemmatized) tokens; scanning unprepared text against it fails silently -- near-zero matches, no error. Both steps therefore share :mod:taters.text.ngram_prep, and the pipeline recipes drive both steps' lemmatize from one shared variable. (The stoplist needs no such agreement: the vocabulary already encodes it.)
• Longest match wins. Scanning prefers the highest-order n-gram at each position and consumes its tokens: "I study health behaviors" scores "health behaviors" once and "health" zero times, not both. A frequency list counts every window; a DTM must not double-count nested terms.

build_doc_term_matrix

build_doc_term_matrix(
    *,
    freq_list_csv,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    on_progress=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    lemmatize=False,
    pos_tagged=False,
    engine="nltk",
    tokenizer="potts",
    stanza_lang="en",
    keep_punctuation=False,
    device="auto",
    weighting="count",
    vocab_min_freq=0,
    vocab_min_obs_pct=0,
    vocab_rule="top_n",
    vocab_top_n=500,
    vocab_rank_by="frequency",
    rounding=4
)

Score every document against a frequency-list vocabulary; write a wide CSV.

Parameters:

Name	Type	Description	Default
freq_list_csv	PathLike	A frequency list written by :func:taters.text.analyze_ngram_frequencies.analyze_ngram_frequencies. In a pipeline this is wired automatically from that step's output.	required
out_features_csv	str or Path	Output file path. If None, defaults to ./features/doc_term_matrix/<analysis_ready_stem>_<weighting>.csv -- the weighting is part of the name, so rerunning with a different one writes a sibling file instead of skipping or clobbering.	None
overwrite_existing	bool	If False and the output file already exists, skip processing and return the path. Also controls whether the intermediate analysis-ready CSV is rebuilt from the current source or reused.	False
encoding	str	Encoding for reading and writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	When gathering from a CSV, name(s) of the column(s) containing text.	("text",)
id_cols	Sequence[str] or None	Optional ID columns that identify each row when gathering from CSV.	None
mode	('concat', 'separate')	Gathering behavior when multiple text columns are provided: "concat" joins them into one text per row; "separate" measures each column on its own.	"concat"
group_by	Sequence[str] or None	Optional grouping keys used during CSV gathering (e.g. ["speaker"]) -- one document per group instead of one per row.	None
delimiter	str	Column separator of the input CSV.	","
pattern	str	Which files to read when gathering from a folder of documents (globs, ;-separated). Only used with txt_dir.	every document type
lemmatize	bool	Must match the frequency list's setting -- the pipeline drives both from one shared variable for exactly this reason.	False
pos_tagged	bool	Must also match the frequency list (checked against its pos column, both ways). Tagged term columns read "felt (VBD)".	False
engine	('nltk', 'stanza')	Who tags: NLTK's perceptron tagger, or Stanza's neural pipeline. Stanza is slower, more accurate, multilingual, and GPU-optional.	"nltk"
tokenizer	('potts', 'stanza')	Who splits text into tokens. The default (the Potts social-media tokenizer) keeps counts comparable across engines and keeps emoticons, hashtags and URLs whole; "stanza" (Stanza engine only) hands Stanza the whole job.	"potts"
stanza_lang	str	Language for the Stanza engine; that language's model is downloaded once on first use (can be a few hundred MB). Only with engine="stanza".	"en"
keep_punctuation	bool	Count punctuation (and emoticons) as terms. Off, only tokens with a letter or digit are counted, so "." and "," never become vocabulary. Must match across the frequency list, the matrix and the topic model.	False
device	('auto', 'cuda', 'cpu')	Where Stanza runs: "auto", "cuda", or "cpu". Only with engine="stanza"; the NLTK engine is CPU-only either way.	"auto"
weighting	('count', 'binary', 'relfreq', 'tfidf')	Cell values: raw matches; 0/1; matches over the document's token count; or matches times the term's IDF from the frequency list. Counts are the neutral default and what topic models want.	"count"
vocab_rule	('top_n', 'min_obs_pct', 'min_freq')	Which single rule decides the vocabulary. Exactly one applies, and the setting below that belongs to it is the only one read: top_n -- keep the vocab_top_n highest-ranked terms. The usual choice: it fixes the width of the matrix, so you know what you are getting whatever the corpus looks like. min_obs_pct -- keep terms appearing in at least vocab_min_obs_pct percent of documents. Use it when you want terms that are widespread rather than merely common, which is the right idea for a topic model. min_freq -- keep terms used at least vocab_min_freq times in total.	"top_n"
vocab_top_n	int	How many terms to keep, when vocab_rule is "top_n". Zero means no limit. Ties at the cut-off are kept, so the choice between two equally-ranked terms is never arbitrary.	500
vocab_min_obs_pct	float	The percentage of documents a term must appear in, when vocab_rule is "min_obs_pct".	0
vocab_min_freq	float	The total number of uses a term must have, when vocab_rule is "min_freq".	0
vocab_rank_by	('frequency', 'obs_pct')	What top_n ranks by. Ignored by the other two rules.	"frequency"
workers	int	Parallel processes for reading documents. 0 means automatic: three-quarters of the logical cores; 1 turns parallelism off. Output files are identical whatever the worker count.	0
rounding	int	Decimal places for the derived values.	4

Returns:

Type	Description
Path	out_features_csv: text_id, token_count, then one column per term, in descending frequency-list order.

Source code in src\taters\text\build_doc_term_matrix.py

@records_settings(
    # `freq_list_csv` is the vocabulary we scanned this matrix against, and
    # the chain walk follows it: the frequency list's own settings (lemmatize,
    # the stoplists, the vocabulary rule) decide these numbers, and we don't
    # restate them here.
    # ... and it's an *asset* rather than a binding, because the vocabulary is
    # the instrument: two matrices scanned against different lists have
    # different columns, and a model fitted on one can't be scored on the
    # other. since we record it by content, the list travels inside any model
    # fitted on this matrix, and a later corpus gets scanned against the same
    # words
    binding=TEXT_INPUT, grain=TEXT_GRAIN,
    assets={"freq_list_csv": None},
    outputs=("out_features_csv",),
    # how many tokens the row had is just bookkeeping, not a term count
    bookkeeping=("token_count",))
def build_doc_term_matrix(
    *,
    # ----- The vocabulary -----
    freq_list_csv: PathLike,

    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    on_progress: Optional[Callable[[int, int], None]] = None,

    # ----- Output -----
    out_features_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    workers: int = 0,

    # ====== SHARED I/O OPTIONS ======
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS (used when csv_path is provided) ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ====== TXT FOLDER GATHER OPTIONS (used when txt_dir is provided) ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ====== DTM OPTIONS ======
    lemmatize: bool = False,
    pos_tagged: bool = False,
    engine: Literal["nltk", "stanza"] = "nltk",
    tokenizer: Literal["potts", "stanza"] = "potts",
    stanza_lang: str = "en",
    keep_punctuation: bool = False,
    device: str = "auto",
    weighting: Literal["count", "binary", "relfreq", "tfidf"] = "count",
    vocab_min_freq: float = 0,
    vocab_min_obs_pct: float = 0,
    vocab_rule: str = "top_n",
    vocab_top_n: int = 500,
    vocab_rank_by: Literal["frequency", "obs_pct"] = "frequency",
    rounding: int = 4,
) -> Path:
    """
    Score every document against a frequency-list vocabulary; write a wide CSV.

    Parameters
    ----------
    freq_list_csv
        A frequency list written by
        :func:`taters.text.analyze_ngram_frequencies.analyze_ngram_frequencies`.
        In a pipeline this is wired automatically from that step's output.
    out_features_csv : str or pathlib.Path, optional
        Output file path. If ``None``, defaults to
        ``./features/doc_term_matrix/<analysis_ready_stem>_<weighting>.csv``
        -- the weighting is part of the name, so rerunning with a different
        one writes a sibling file instead of skipping or clobbering.
    overwrite_existing : bool, default=False
        If ``False`` and the output file already exists, skip processing and
        return the path. Also controls whether the intermediate analysis-ready
        CSV is rebuilt from the current source or reused.
    encoding : str, default="utf-8-sig"
        Encoding for reading and writing CSV files.
    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV, name(s) of the column(s) containing text.
    id_cols : Sequence[str] or None, optional
        Optional ID columns that identify each row when gathering from CSV.
    mode : {"concat", "separate"}, default="concat"
        Gathering behavior when multiple text columns are provided:
        ``"concat"`` joins them into one text per row; ``"separate"`` measures
        each column on its own.
    group_by : Sequence[str] or None, optional
        Optional grouping keys used during CSV gathering (e.g. ``["speaker"]``)
        -- one document per group instead of one per row.
    delimiter : str, default=","
        Column separator of the *input* CSV.
    pattern : str, default=every document type
        Which files to read when gathering from a folder of documents
        (globs, ``;``-separated). Only used with ``txt_dir``.
    lemmatize : bool, default=False
        Must match the frequency list's setting -- the pipeline drives both
        from one shared variable for exactly this reason.
    pos_tagged : bool, default=False
        Must also match the frequency list (checked against its ``pos``
        column, both ways). Tagged term columns read "felt (VBD)".
    engine : {"nltk", "stanza"}, default="nltk"
        Who tags: NLTK's perceptron tagger, or Stanza's neural pipeline.
        Stanza is slower, more accurate, multilingual, and GPU-optional.
    tokenizer : {"potts", "stanza"}, default="potts"
        Who splits text into tokens. The default (the Potts social-media
        tokenizer) keeps counts comparable across engines and keeps
        emoticons, hashtags and URLs whole; "stanza" (Stanza engine only)
        hands Stanza the whole job.
    stanza_lang : str, default="en"
        Language for the Stanza engine; that language's model is downloaded
        once on first use (can be a few hundred MB). Only with engine="stanza".
    keep_punctuation : bool, default=False
        Count punctuation (and emoticons) as terms. Off, only tokens with a
        letter or digit are counted, so "." and "," never become vocabulary.
        Must match across the frequency list, the matrix and the topic model.
    device : {"auto", "cuda", "cpu"}, default="auto"
        Where Stanza runs: "auto", "cuda", or "cpu". Only with engine="stanza";
        the NLTK engine is CPU-only either way.
    weighting : {"count", "binary", "relfreq", "tfidf"}, default="count"
        Cell values: raw matches; 0/1; matches over the document's token
        count; or matches times the term's IDF from the frequency list.
        Counts are the neutral default and what topic models want.
    vocab_rule : {"top_n", "min_obs_pct", "min_freq"}, default="top_n"
        Which single rule decides the vocabulary. Exactly one applies, and
        the setting below that belongs to it is the only one read:

        * ``top_n`` -- keep the ``vocab_top_n`` highest-ranked terms.
          The usual choice: it fixes the width of the matrix, so you know
          what you are getting whatever the corpus looks like.
        * ``min_obs_pct`` -- keep terms appearing in at least
          ``vocab_min_obs_pct`` percent of documents. Use it when you want
          terms that are *widespread* rather than merely common, which is
          the right idea for a topic model.
        * ``min_freq`` -- keep terms used at least ``vocab_min_freq`` times
          in total.
    vocab_top_n : int, default=500
        How many terms to keep, when ``vocab_rule`` is ``"top_n"``. Zero
        means no limit. Ties at the cut-off are kept, so the choice between
        two equally-ranked terms is never arbitrary.
    vocab_min_obs_pct : float, default=0
        The percentage of documents a term must appear in, when
        ``vocab_rule`` is ``"min_obs_pct"``.
    vocab_min_freq : float, default=0
        The total number of uses a term must have, when ``vocab_rule`` is
        ``"min_freq"``.
    vocab_rank_by : {"frequency", "obs_pct"}, default="frequency"
        What ``top_n`` ranks by. Ignored by the other two rules.
    workers : int, default=0
        Parallel processes for reading documents. ``0`` means automatic:
        three-quarters of the logical cores; ``1`` turns parallelism off. Output files are
        identical whatever the worker count.
    rounding : int, default=4
        Decimal places for the derived values.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id``, ``token_count``, then one column
        per term, in descending frequency-list order.
    """
    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        # the weighting goes in the FILENAME, not just the cells. with a bare
        # name, rerunning with a different weighting either handed back the
        # old matrix untouched (overwrite off -- silently the wrong numbers)
        # or destroyed it (overwrite on). this way the four matrices of one
        # corpus can all live side by side
        out_features_csv = (Path.cwd() / "features" / "doc_term_matrix"
                            / f"{analysis_ready.stem}_{weighting}.csv")
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)

    if not overwrite_existing and out_features_csv.is_file():
        print("Doc-term matrix output file already exists; returning existing file.")
        return out_features_csv

    vocab = _load_vocabulary(
        Path(freq_list_csv),
        encoding=encoding,
        pos_tagged=pos_tagged,
        vocab_min_freq=vocab_min_freq,
        vocab_min_obs_pct=vocab_min_obs_pct,
        vocab_rule=vocab_rule,
        vocab_top_n=vocab_top_n,
        vocab_rank_by=vocab_rank_by,
    )
    # column order: descending frequency, ties alphabetical -- same order the
    # frequency list is written in, so that the two files read side by side
    terms = sorted(vocab,
                   key=lambda g: (-vocab[g]["frequency"], words_of(g), tags_of(g)))
    term_index = {g: i for i, g in enumerate(terms)}
    max_n = max(g.count(" ") + 1 for g in terms)

    columns = column_names(terms, pos_tagged)
    idf = [vocab[g]["idf"] for g in terms]

    if engine == "stanza" and (lemmatize or pos_tagged or tokenizer == "stanza"):
        announce(on_progress, "loading the stanza pipeline (first use "
                              "downloads its model)")
    stream = make_token_stream(lemmatize, pos_tagged, engine=engine,
                               tokenizer=tokenizer,
                               stanza_lang=stanza_lang, device=device,
                               keep_punctuation=keep_punctuation)

    # 2) scan and write, one document per row, on the shared pooled-row
    #    driver. that gives us results in file order no matter the worker
    #    count, plus one live sub-bar per document in flight. stanza-backed
    #    streams stay single-process (see `pooled_text_workers`)
    from ..helpers.row_map import map_text_rows
    from .ngram_prep import pooled_text_workers

    stream_args = dict(lemmatize=lemmatize, pos_tagged=pos_tagged,
                       engine=engine, tokenizer=tokenizer,
                       stanza_lang=stanza_lang, device=device,
                       keep_punctuation=keep_punctuation)
    with atomic_write(out_features_csv, newline="", encoding=encoding) as out:
        writer = csv.writer(out)
        writer.writerow(["text_id", "token_count", *columns])
        for row, (n_tokens, scores) in map_text_rows(
                analysis_ready, encoding=encoding,
                workers=lambda n_rows: pooled_text_workers(
                    workers, n_rows, engine=engine, tokenizer=tokenizer,
                    lemmatize=lemmatize, pos_tagged=pos_tagged),
                message="building the matrix", on_progress=on_progress,
                inline_fn=lambda pair: _cells_of(
                    stream, term_index, max_n, weighting, idf, rounding,
                    pair[1]),
                pool_fn=_scan_in_worker,
                initializer=_init_scan_worker,
                initargs=(stream_args, terms, weighting, idf, rounding)):
            writer.writerow([row.get("text_id", ""), n_tokens, *scores])

    return out_features_csv

column_names

column_names(terms, pos_tagged)

Display names for the term columns, unique after the two reserved ones.

"text_id" is a perfectly plausible token, and left as-is it duplicated the header -- reading the file back by name then returned the term's count where the document id should be. Colliding terms get a trailing underscore (repeatedly, in case that name is somehow taken too). Tagged terms read "felt (VBD)", which also keeps the verb and the noun apart as columns.

Source code in src\taters\text\build_doc_term_matrix.py

def column_names(terms: Sequence[str], pos_tagged: bool) -> list:
    """
    Display names for the term columns, unique after the two reserved ones.

    "text_id" is a perfectly plausible *token*, and left as-is it duplicated
    the header -- reading the file back by name then returned the term's count
    where the document id should be. Colliding terms get a trailing underscore
    (repeatedly, in case that name is somehow taken too). Tagged terms read
    "felt (VBD)", which also keeps the verb and the noun apart as columns.
    """
    used = {"text_id", "token_count"}
    columns = []
    for gram in terms:
        name = f"{words_of(gram)} ({tags_of(gram)})" if pos_tagged else gram
        while name in used:
            name += "_"
        used.add(name)
        columns.append(name)
    return columns

scan_tokens

scan_tokens(tokens, term_index, max_n)

Raw match counts for one document, longest match first.

Matched tokens are consumed, so a nested term is not also counted ("health behaviors" beats "health"). The C# plugin's window guard was off by one and could never match an n-gram that ran to the end of the document; i + n <= len can. Shared with the MEM topic model's apply path, so a saved model scores new text with the same scan, not a copy.

Source code in src\taters\text\build_doc_term_matrix.py

def scan_tokens(tokens: Sequence[str], term_index: Dict[str, int],
                max_n: int) -> list:
    """
    Raw match counts for one document, longest match first.

    Matched tokens are consumed, so a nested term is not also counted
    ("health behaviors" beats "health"). The C# plugin's window guard was off
    by one and could never match an n-gram that ran to the end of the
    document; ``i + n <= len`` can. Shared with the MEM topic model's apply
    path, so a saved model scores new text with the *same* scan, not a copy.
    """
    scores = [0.0] * len(term_index)
    i = 0
    while i < len(tokens):
        for n in range(min(max_n, len(tokens) - i), 0, -1):
            gram = " ".join(tokens[i:i + n])
            idx = term_index.get(gram)
            if idx is not None:
                scores[idx] += 1
                i += n
                break
        else:
            i += 1
    return scores

weight_scores

weight_scores(scores, weighting, n_tokens, idf, rounding)

Turn raw match counts into the requested cell values.

Source code in src\taters\text\build_doc_term_matrix.py

def weight_scores(scores: list, weighting: str, n_tokens: int,
                  idf: Sequence[float], rounding: int) -> list:
    """Turn raw match counts into the requested cell values."""
    if weighting == "binary":
        return [1 if s else 0 for s in scores]
    if weighting == "relfreq":
        return [round(s / n_tokens, rounding) if n_tokens else 0
                for s in scores]
    if weighting == "tfidf":
        return [round(s * idf[j], rounding) for j, s in enumerate(scores)]
    return [int(s) for s in scores]

taters.text.analyze_parts_of_speech

Part-of-speech features, one row per document -- the content coder's shape.

Each document is tokenized (happierfuntokenizing, the same tokenizer as the n-gram tools), POS-tagged with NLTK, and summarized as one column per tag: relative frequencies by default, raw counts on request. Syntactic n-grams -- sequences of tags, "DT_NN", "PRP_VBD_JJ" -- are available the same way: all orders from 1 up to sngram_n, each order normalized by its own number of windows in the document, so within a document every order's columns sum to 1.

Two tag sets: Penn Treebank (tagset="penn", NLTK's native ~45 tags) and the Universal tagset (12 coarse categories: NOUN, VERB, ADJ, ...), which is often the better unit for psychological work.

The tagged text itself can also be written out (tagged_text_csv), one row per document with tokens as word_TAG -- off by default, for the rare downstream that wants the tags rather than the summary.

analyze_parts_of_speech

analyze_parts_of_speech(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    on_progress=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    tagset="penn",
    engine="nltk",
    tokenizer="potts",
    stanza_lang="en",
    device="auto",
    relative_freq=True,
    sngram_n=1,
    tagged_text_csv=None,
    rounding=4
)

Tag every document and write one row of POS features per document.

Parameters:

Name	Type	Description	Default
out_features_csv	str or Path	Output file path. If None, defaults to ./features/pos/<analysis_ready_filename>.	None
overwrite_existing	bool	If False and the output file already exists, skip processing and return the path. Also controls whether the intermediate analysis-ready CSV is rebuilt from the current source or reused.	False
encoding	str	Encoding for reading and writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	When gathering from a CSV, name(s) of the column(s) containing text.	("text",)
id_cols	Sequence[str] or None	Optional ID columns that identify each row when gathering from CSV.	None
mode	('concat', 'separate')	Gathering behavior when multiple text columns are provided: "concat" joins them into one text per row; "separate" measures each column on its own.	"concat"
group_by	Sequence[str] or None	Optional grouping keys used during CSV gathering (e.g. ["speaker"]) -- one document per group instead of one per row.	None
delimiter	str	Column separator of the input CSV.	","
pattern	str	Which files to read when gathering from a folder of documents (globs, ;-separated). Only used with txt_dir.	every document type
tagset	('penn', 'universal')	Penn Treebank's ~45 tags, or the Universal tagset's 12 coarse ones.	"penn"
engine	('nltk', 'stanza')	Who tags: NLTK's perceptron tagger, or Stanza's neural pipeline. Stanza is slower, more accurate, multilingual, and GPU-optional.	"nltk"
tokenizer	('potts', 'stanza')	Who splits text into tokens. The default (the Potts social-media tokenizer) keeps counts comparable across engines and keeps emoticons, hashtags and URLs whole; "stanza" (Stanza engine only) hands Stanza the whole job.	"potts"
stanza_lang	str	Language for the Stanza engine; that language's model is downloaded once on first use (can be a few hundred MB). Only with engine="stanza".	"en"
device	('auto', 'cuda', 'cpu')	Where Stanza runs: "auto", "cuda", or "cpu". Only with engine="stanza"; the NLTK engine is CPU-only either way.	"auto"
relative_freq	bool	Each order-n column is that sequence's count divided by the number of length-n windows in the document, so an order's columns sum to 1 per document. False writes raw counts.	True
sngram_n	int	Highest syntactic-n-gram order. All orders from 1 up are written; an order-2 column reads pos_DT_NN.	1
tagged_text_csv	path	Also write the tagged text itself -- text_id, tagged_text with tokens as word_TAG -- for the rare downstream that wants the tags rather than the summary. Off (None) by default.	None
workers	int	Parallel processes for reading documents. 0 means automatic: three-quarters of the logical cores; 1 turns parallelism off. Output files are identical whatever the worker count.	0
rounding	int	Decimal places for the derived values.	4

Returns:

Type	Description
Path	out_features_csv: text_id, any pass-through id_cols, token_count, then one pos_* column per observed tag sequence (orders ascending, alphabetical within an order).

Source code in src\taters\text\analyze_parts_of_speech.py

@records_settings(binding=TEXT_INPUT, grain=TEXT_GRAIN,
                  # `tagged_text_csv` is a second output path, not a setting:
                  # it just says where the tagged text goes
                  outputs=("out_features_csv", "tagged_text_csv"),
                  # our columns are whichever tags (and tag n-grams) actually
                  # show up. so, a tag that no text in a second corpus uses
                  # has no column there, and a model that knows that tag
                  # should read it as zero -- because that's what the count
                  # was. if we refused instead, a trained model couldn't score
                  # a new study just because one particle column was missing
                  absent_means_zero=True,
                  bookkeeping=("token_count",))
def analyze_parts_of_speech(
    *,
    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    on_progress: Optional[Callable[[int, int], None]] = None,

    # ----- Output -----
    out_features_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    workers: int = 0,

    # ====== SHARED I/O OPTIONS ======
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS (used when csv_path is provided) ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ====== TXT FOLDER GATHER OPTIONS (used when txt_dir is provided) ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ====== POS OPTIONS ======
    tagset: Literal["penn", "universal"] = "penn",
    engine: Literal["nltk", "stanza"] = "nltk",
    tokenizer: Literal["potts", "stanza"] = "potts",
    stanza_lang: str = "en",
    device: str = "auto",
    relative_freq: bool = True,
    sngram_n: int = 1,
    tagged_text_csv: Optional[PathLike] = None,
    rounding: int = 4,
) -> Path:
    """
    Tag every document and write one row of POS features per document.

    Parameters
    ----------
    out_features_csv : str or pathlib.Path, optional
        Output file path. If ``None``, defaults to
        ``./features/pos/<analysis_ready_filename>``.
    overwrite_existing : bool, default=False
        If ``False`` and the output file already exists, skip processing and
        return the path. Also controls whether the intermediate analysis-ready
        CSV is rebuilt from the current source or reused.
    encoding : str, default="utf-8-sig"
        Encoding for reading and writing CSV files.
    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV, name(s) of the column(s) containing text.
    id_cols : Sequence[str] or None, optional
        Optional ID columns that identify each row when gathering from CSV.
    mode : {"concat", "separate"}, default="concat"
        Gathering behavior when multiple text columns are provided:
        ``"concat"`` joins them into one text per row; ``"separate"`` measures
        each column on its own.
    group_by : Sequence[str] or None, optional
        Optional grouping keys used during CSV gathering (e.g. ``["speaker"]``)
        -- one document per group instead of one per row.
    delimiter : str, default=","
        Column separator of the *input* CSV.
    pattern : str, default=every document type
        Which files to read when gathering from a folder of documents
        (globs, ``;``-separated). Only used with ``txt_dir``.
    tagset : {"penn", "universal"}, default="penn"
        Penn Treebank's ~45 tags, or the Universal tagset's 12 coarse ones.
    engine : {"nltk", "stanza"}, default="nltk"
        Who tags: NLTK's perceptron tagger, or Stanza's neural pipeline.
        Stanza is slower, more accurate, multilingual, and GPU-optional.
    tokenizer : {"potts", "stanza"}, default="potts"
        Who splits text into tokens. The default (the Potts social-media
        tokenizer) keeps counts comparable across engines and keeps
        emoticons, hashtags and URLs whole; "stanza" (Stanza engine only)
        hands Stanza the whole job.
    stanza_lang : str, default="en"
        Language for the Stanza engine; that language's model is downloaded
        once on first use (can be a few hundred MB). Only with engine="stanza".
    device : {"auto", "cuda", "cpu"}, default="auto"
        Where Stanza runs: "auto", "cuda", or "cpu". Only with engine="stanza";
        the NLTK engine is CPU-only either way.
    relative_freq : bool, default=True
        Each order-n column is that sequence's count divided by the number of
        length-n windows in the document, so an order's columns sum to 1 per
        document. False writes raw counts.
    sngram_n : int, default=1
        Highest syntactic-n-gram order. All orders from 1 up are written;
        an order-2 column reads ``pos_DT_NN``.
    tagged_text_csv : path, optional
        Also write the tagged text itself -- ``text_id, tagged_text`` with
        tokens as ``word_TAG`` -- for the rare downstream that wants the tags
        rather than the summary. Off (None) by default.
    workers : int, default=0
        Parallel processes for reading documents. ``0`` means automatic:
        three-quarters of the logical cores; ``1`` turns parallelism off. Output files are
        identical whatever the worker count.
    rounding : int, default=4
        Decimal places for the derived values.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id``, any pass-through ``id_cols``,
        ``token_count``, then one ``pos_*`` column per observed tag sequence
        (orders ascending, alphabetical within an order).
    """
    if sngram_n < 1:
        raise ValueError(f"sngram_n must be >= 1, got {sngram_n}")

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        out_features_csv = Path.cwd() / "features" / "pos" / analysis_ready.name
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)

    if not overwrite_existing and out_features_csv.is_file():
        print("POS features output file already exists; returning existing file.")
        return out_features_csv

    if engine == "stanza":
        announce(on_progress, "loading the stanza pipeline (first use "
                              "downloads its model)")
    tag = make_tagged_stream(engine=engine, tokenizer=tokenizer,
                             tagset=tagset, stanza_lang=stanza_lang,
                             device=device)   # this is the inline path's tagger


    # 2) one pooled pass: tag and count, with results in file order and one
    #    live sub-bar per document in flight. we hold the rows in memory
    #    afterwards because the column set is the union of every sequence any
    #    document produced -- we can't know that until we've read the last
    #    document. stanza tagging stays single-process (see
    #    `pooled_text_workers`); NLTK pools just fine
    from ..helpers.row_map import map_text_rows
    from .ngram_prep import pooled_text_workers

    # there's one shared rule for what rides along beside text_id -- see
    # resolve_passthrough_columns. it also does the header check that
    # map_text_rows can't do for us
    from ..helpers.row_map import resolve_passthrough_columns

    with analysis_ready.open("r", newline="", encoding=encoding) as f:
        fields = csv.DictReader(f).fieldnames or []
    passthrough = resolve_passthrough_columns(
        fields, id_cols=id_cols, group_by=group_by,
        analysis_ready=analysis_ready)

    tag_args = dict(engine=engine, tokenizer=tokenizer, tagset=tagset,
                    stanza_lang=stanza_lang, device=device)
    docs: List[tuple] = []       # (text_id, meta, token_count, Counter)
    tagged_rows: List[tuple] = []
    for row, (n_tags, counts, tagged_text) in map_text_rows(
            analysis_ready, encoding=encoding,
            workers=lambda n_rows: pooled_text_workers(
                workers, n_rows, engine=engine, tokenizer=tokenizer,
                lemmatize=False, pos_tagged=True),
            message="tagging documents", on_progress=on_progress,
            inline_fn=lambda pair: _tag_counts(
                tag, sngram_n, tagged_text_csv is not None, pair[1]),
            pool_fn=_tags_in_worker,
            initializer=_init_tag_worker,
            initargs=(tag_args, sngram_n, tagged_text_csv is not None)):
        meta = {c: str(row.get(c, "") or "") for c in passthrough}
        docs.append((str(row.get("text_id", "") or ""), meta, n_tags, counts))
        if tagged_text_csv is not None:
            tagged_rows.append((str(row.get("text_id", "") or ""), tagged_text))

    # 3) the column plan: orders ascending, alphabetical within an order
    seen = sorted({key for *_ , counts in docs for key in counts})
    columns = [(n, gram, "pos_" + gram.replace(" ", "_")) for n, gram in seen]

    with atomic_write(out_features_csv, newline="", encoding=encoding) as f:
        writer = csv.writer(f)
        writer.writerow(["text_id", *passthrough, "token_count",
                         *(name for *_ , name in columns)])
        for text_id, meta, token_count, counts in docs:
            cells = []
            for n, gram, _name in columns:
                value = counts.get((n, gram), 0)
                if relative_freq:
                    windows = max(token_count - n + 1, 0)
                    value = round(value / windows, rounding) if windows else 0
                cells.append(value)
            writer.writerow([text_id, *(meta[c] for c in passthrough),
                             token_count, *cells])

    if tagged_text_csv is not None:
        tagged_text_csv = Path(tagged_text_csv)
        tagged_text_csv.parent.mkdir(parents=True, exist_ok=True)
        with atomic_write(tagged_text_csv, newline="", encoding=encoding) as f:
            writer = csv.writer(f)
            writer.writerow(["text_id", "tagged_text"])
            writer.writerows(tagged_rows)

    return out_features_csv

taters.text.analyze_cohesion

Text cohesion features (TAACO-style), one row per document.

The measure families of Crossley, Kyle & McNamara's TAACO (2016) and the Coh-Metrix lineage behind it (Graesser, McNamara, Louwerse & Cai 2004), reimplemented from their published definitions: lexical overlap between adjacent sentences and paragraphs across nine word classes, lemma-based type-token ratios and lexical density, and givenness. Column names follow TAACO 2.1.3 wherever the measure survives, so results can be read against the TAACO literature.

This is a REIMPLEMENTATION, not a port. The reference implementation (reference/TAACO-main, CC BY-NC-SA -- nothing was copied) was audited line by line and several of its defects are deliberately not reproduced. Where behavior differs, the code comments say exactly what TAACO did and what this does instead, and COHESION_MEASURES.md (shipped next to this module) documents every column: formula, range, interpretation, lineage, and differences. The headline corrections:

• Documents with too few segments emit empty cells (NA), not 0.0 -- TAACO scores a one-paragraph essay as "zero paragraph cohesion", which poisons any downstream average (5.6% of its own sample corpus).
• Real punctuation filtering: a token must contain a letter or digit. TAACO's punctuation list mixed POS tags with literal characters that can never match a tag, so % (Penn-tagged NN) counted as a noun in every noun, content and argument index, and stray quotes/hyphens inflated the word counts underneath every ratio.
• The two-segment windows are built by concatenation into fresh lists. TAACO appended the third segment into its shared sentence list, so computing one index corrupted the input of the next -- its published values depend on which checkboxes were ticked.

adjacent_semantic

adjacent_semantic(embeddings)

Mean adjacent cosine over row-normalized segment embeddings: window one (segment i vs i+1) and window two (segment i vs the normalized mean of i+1 and i+2 -- a fresh vector, never TAACO's in-place concatenation of shared state, audit finding 4.2). Too few segments: None.

Source code in src\taters\text\analyze_cohesion.py

def adjacent_semantic(embeddings) -> Tuple[Optional[float], Optional[float]]:
    """
    Mean adjacent cosine over row-normalized segment embeddings: window one
    (segment i vs i+1) and window two (segment i vs the normalized mean of
    i+1 and i+2 -- a fresh vector, never TAACO's in-place concatenation of
    shared state, audit finding 4.2). Too few segments: None.
    """
    import numpy as np

    E = np.asarray(embeddings, dtype=np.float64)
    n = len(E)
    if n < 2:
        return None, None
    sim1 = float(np.mean([E[i] @ E[i + 1] for i in range(n - 1)]))
    if n < 3:
        return sim1, None
    sims2 = []
    for i in range(n - 2):
        pair = E[i + 1] + E[i + 2]
        norm = np.linalg.norm(pair)
        sims2.append(E[i] @ (pair / norm) if norm else 0.0)
    return sim1, float(np.mean(sims2))

analyze_cohesion

analyze_cohesion(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    on_progress=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    engine="nltk",
    tokenizer="potts",
    stanza_lang="en",
    device="auto",
    mattr_window=50,
    connective_lists=None,
    semantic_model=DEFAULT_SEMANTIC_MODEL,
    rounding=4
)

Compute TAACO-style cohesion indices; one row per document.

See COHESION_MEASURES.md (shipped next to this module) for what every column measures, how to interpret it, and where this implementation deliberately differs from TAACO 2.1.3.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
txt_dir	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
analysis_csv	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
gathered_csv	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
out_features_csv	str or Path	Output file path. If None, defaults to ./features/cohesion/<analysis_ready_filename>.	None
overwrite_existing	bool	If False and the output file already exists, skip processing and return the path.	False
encoding	str	Encoding for reading and writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	When gathering from a CSV, name(s) of the column(s) containing text.	("text",)
id_cols	Sequence[str] or None	Optional ID columns that identify each row when gathering from CSV.	None
mode	('concat', 'separate')	Gathering behavior when multiple text columns are provided.	"concat"
group_by	Sequence[str] or None	Optional grouping keys used during CSV gathering.	None
delimiter	str	Column separator of the input CSV.	","
pattern	str	Which files to read when gathering from a folder of documents.	every document type
engine	('nltk', 'stanza')	Who tags and lemmatizes: NLTK (fast, English), or Stanza (neural, multilingual, GPU-optional). Sentence splitting is punkt under NLTK and the model's own under Stanza.	"nltk"
tokenizer	('potts', 'stanza')	Who splits text into tokens, exactly as in the n-gram steps.	"potts"
stanza_lang	str	Language for the Stanza engine; the model downloads once on first use. Only with engine="stanza".	"en"
device	('auto', 'cuda', 'cpu')	Where Stanza runs: "auto", "cuda", or "cpu". Only with engine="stanza".	"auto"
mattr_window	int	Window (tokens) for the moving-average TTRs. TAACO hardcodes 50; this default matches, and shorter documents fall back to plain TTR.	50
connective_lists	sequence of paths	Connectives category files (.txt, one word or phrase per line, # comments, optional TAB + Penn tag constraint; folders are expanded). Each file becomes one incidence column named by its stem. None (the default) means the shipped categories; in a pipeline, the library's connectives shelf is wired in, so lists you import there become columns automatically.	None
semantic_model	str	The embedding model for the four semantic-similarity columns (adjacent sentence/paragraph cosine). "none" skips them and the step runs without any model. Replaces TAACO's frozen COCA LSA and its LDA index (the latter provably broken in the reference tool).	the MiniLM sentence-transformer
workers	int	Parallel processes for reading and measuring documents. 0 means automatic: three-quarters of the logical cores. Output files are identical whatever the worker count; Stanza-backed runs stay single-process (its lever is batching, not processes).	0
rounding	int	Decimal places for every index.	4

Returns:

Type	Description
Path	out_features_csv: text_id, nwords, then one column per index (~130). Indices that are undefined for a document -- paragraph cohesion of a one-paragraph text -- are empty cells, never 0.

Source code in src\taters\text\analyze_cohesion.py

@records_settings(binding=TEXT_INPUT, grain=TEXT_GRAIN,
                  outputs=("out_features_csv",),
                  bookkeeping=("nwords",),
                  assets={"connective_lists": "connectives"})
def analyze_cohesion(
    *,
    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    on_progress: Optional[Callable[[int, int], None]] = None,

    # ----- Output -----
    out_features_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    workers: int = 0,

    # ====== SHARED I/O OPTIONS ======
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS (used when csv_path is provided) ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ====== TXT FOLDER GATHER OPTIONS (used when txt_dir is provided) ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ====== COHESION OPTIONS ======
    engine: Literal["nltk", "stanza"] = "nltk",
    tokenizer: Literal["potts", "stanza"] = "potts",
    stanza_lang: str = "en",
    device: str = "auto",
    mattr_window: int = 50,
    connective_lists: Optional[Sequence[PathLike]] = None,
    semantic_model: str = DEFAULT_SEMANTIC_MODEL,
    rounding: int = 4,
) -> Path:
    """
    Compute TAACO-style cohesion indices; one row per document.

    See ``COHESION_MEASURES.md`` (shipped next to this module) for what every
    column measures, how to interpret it, and where this implementation
    deliberately differs from TAACO 2.1.3.

    Parameters
    ----------
    csv_path, txt_dir, analysis_csv, gathered_csv
        The same input contract as the other text analyzers: a spreadsheet of
        texts, a folder of documents, or a prebuilt analysis-ready CSV.
    out_features_csv : str or pathlib.Path, optional
        Output file path. If ``None``, defaults to
        ``./features/cohesion/<analysis_ready_filename>``.
    overwrite_existing : bool, default=False
        If ``False`` and the output file already exists, skip processing and
        return the path.
    encoding : str, default="utf-8-sig"
        Encoding for reading and writing CSV files.
    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV, name(s) of the column(s) containing text.
    id_cols : Sequence[str] or None, optional
        Optional ID columns that identify each row when gathering from CSV.
    mode : {"concat", "separate"}, default="concat"
        Gathering behavior when multiple text columns are provided.
    group_by : Sequence[str] or None, optional
        Optional grouping keys used during CSV gathering.
    delimiter : str, default=","
        Column separator of the *input* CSV.
    pattern : str, default=every document type
        Which files to read when gathering from a folder of documents.
    engine : {"nltk", "stanza"}, default="nltk"
        Who tags and lemmatizes: NLTK (fast, English), or Stanza (neural,
        multilingual, GPU-optional). Sentence splitting is punkt under NLTK
        and the model's own under Stanza.
    tokenizer : {"potts", "stanza"}, default="potts"
        Who splits text into tokens, exactly as in the n-gram steps.
    stanza_lang : str, default="en"
        Language for the Stanza engine; the model downloads once on first
        use. Only with engine="stanza".
    device : {"auto", "cuda", "cpu"}, default="auto"
        Where Stanza runs: "auto", "cuda", or "cpu". Only with
        engine="stanza".
    mattr_window : int, default=50
        Window (tokens) for the moving-average TTRs. TAACO hardcodes 50;
        this default matches, and shorter documents fall back to plain TTR.
    connective_lists : sequence of paths, optional
        Connectives category files (.txt, one word or phrase per line, ``#``
        comments, optional TAB + Penn tag constraint; folders are expanded).
        Each file becomes one incidence column named by its stem. ``None``
        (the default) means the shipped categories; in a pipeline, the
        library's connectives shelf is wired in, so lists you import there
        become columns automatically.
    semantic_model : str, default the MiniLM sentence-transformer
        The embedding model for the four semantic-similarity columns
        (adjacent sentence/paragraph cosine). ``"none"`` skips them and the
        step runs without any model. Replaces TAACO's frozen COCA LSA and
        its LDA index (the latter provably broken in the reference tool).
    workers : int, default=0
        Parallel processes for reading and measuring documents. ``0`` means
        automatic: three-quarters of the logical cores. Output files are
        identical whatever the worker count; Stanza-backed runs stay
        single-process (its lever is batching, not processes).
    rounding : int, default=4
        Decimal places for every index.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id``, ``nwords``, then one column per
        index (~130). Indices that are undefined for a document -- paragraph
        cohesion of a one-paragraph text -- are empty cells, never 0.
    """
    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        out_features_csv = Path.cwd() / "features" / "cohesion" / analysis_ready.name
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)
    if not overwrite_existing and out_features_csv.is_file():
        print("Cohesion output file already exists; returning existing file.")
        return out_features_csv

    if engine == "stanza":
        # the model might download (hundreds of MB) or load right here, so we
        # say so. otherwise the step just sits on "reading the input" while
        # it's clearly off doing something else
        announce(on_progress, "loading the stanza pipeline (first use "
                              "downloads its model)")
    parse = make_sentence_stream(engine=engine, tokenizer=tokenizer,
                                 stanza_lang=stanza_lang, device=device)
    stream_args = dict(engine=engine, tokenizer=tokenizer,
                       stanza_lang=stanza_lang, device=device)
    connectives = load_connective_lists(connective_lists)
    semantic_on = str(semantic_model).strip().lower() not in ("", "none")
    columns = header_columns([name for name, _ in connectives],
                             semantic=semantic_on)
    encoder = None
    if semantic_on:
        # the model lives HERE, in the parent, exactly once -- the workers stay
        # model-free. our house rule: a model's speed lever is batching, and
        # one copy per worker process is a memory bill nobody wants to pay
        from sentence_transformers import SentenceTransformer

        from ..helpers.gpu import resolve_device

        announce(on_progress, "loading the sentence-embedding model")
        encoder = SentenceTransformer(semantic_model,
                                      device=resolve_device(device)[0])
    insert_at = columns.index("semantic_1_all_sent") if semantic_on else -1

    from ..helpers.row_map import map_text_rows

    def _fmt(value: Optional[float]) -> object:
        # NA is an EMPTY CELL. TAACO writes 0.0 for indices that don't exist
        # for a document, but downstream you can't tell that apart from a
        # measured zero cohesion, so we don't do that
        if value is None:
            return ""
        return round(float(value), rounding)

    with atomic_write(out_features_csv, newline="", encoding=encoding) as out:
        writer = csv.writer(out)
        writer.writerow(["text_id", "nwords", *columns])
        for row, (nwords, values, sent_texts, para_texts) in map_text_rows(
                analysis_ready, encoding=encoding,
                workers=lambda n_rows: pooled_text_workers(
                    workers, n_rows, engine=engine, tokenizer=tokenizer,
                    lemmatize=True, pos_tagged=True),
                message="measuring cohesion", on_progress=on_progress,
                inline_fn=lambda pair: _measure_text(
                    parse, pair[1], mattr_window, connectives, semantic_on),
                pool_fn=_cohesion_in_worker,
                initializer=_init_cohesion_worker,
                initargs=(stream_args, mattr_window, connectives, semantic_on)):
            if encoder is not None:
                semantic = _semantic_values(encoder, sent_texts or [],
                                            para_texts or [])
                # the worker computed the lexical row without the semantic
                # slots (workers are model-free, remember), so we splice them
                # in at their column position. `columns` and the worker's
                # values share the same prefix, so the index in the full header
                # is exactly where they go
                values = values[:insert_at] + semantic + values[insert_at:]
            writer.writerow([row.get("text_id", ""), nwords,
                             *(_fmt(v) for v in values)])
    return out_features_csv

classify_sentence

classify_sentence(triples)

One sentence's (word, lemma, tag) triples -> lemma lists per class.

Class rules (all deviations from TAACO documented):

• noun: NN/NNS/NNP/NNPS -- proper nouns included, as in TAACO.
• pronoun: PRP/PRP$ plus unattended demonstratives -- this/that/these/ those NOT followed by a noun or adjective ("I like that" yes, "that car" no). TAACO used the dependency parse for attendedness; this uses the next content-bearing token's tag, documented as a heuristic. "that" as a complementizer (tag IN) is neither.
• verb: content verbs only, exactly as TAACO's verb indices are -- a modal (MD) or a form of "be" is a function word. TAACO additionally demoted auxiliary "have"/"do" via spaCy's AUX tag; without a parse we keep have/do as content, a documented deviation.
• adv: deadjectival adverbs are content (see _adverb_is_content).
• cw: nouns + adjectives + content verbs + content adverbs. fw: every other counted token.
• argument: nouns + pronouns (TAACO's definition).

Source code in src\taters\text\analyze_cohesion.py

def classify_sentence(triples: Sequence[Tuple[str, str, str]]) -> Dict[str, List[str]]:
    """
    One sentence's ``(word, lemma, tag)`` triples -> lemma lists per class.

    Class rules (all deviations from TAACO documented):

    * noun: NN/NNS/NNP/NNPS -- proper nouns included, as in TAACO.
    * pronoun: PRP/PRP$ plus *unattended demonstratives* -- this/that/these/
      those NOT followed by a noun or adjective ("I like *that*" yes,
      "*that* car" no). TAACO used the dependency parse for attendedness;
      this uses the next content-bearing token's tag, documented as a
      heuristic. "that" as a complementizer (tag IN) is neither.
    * verb: content verbs only, exactly as TAACO's verb indices are -- a
      modal (MD) or a form of "be" is a function word. TAACO additionally
      demoted auxiliary "have"/"do" via spaCy's AUX tag; without a parse we
      keep have/do as content, a documented deviation.
    * adv: deadjectival adverbs are content (see `_adverb_is_content`).
    * cw: nouns + adjectives + content verbs + content adverbs.
      fw: every other counted token.
    * argument: nouns + pronouns (TAACO's definition).
    """
    out: Dict[str, List[str]] = {c: [] for c in CLASS_ORDER}
    n = len(triples)
    for i, (word, lemma, tag) in enumerate(triples):
        if not _ALNUM.search(word):
            continue
        out["all"].append(lemma)

        content = False
        if tag in _NOUN_TAGS:
            out["noun"].append(lemma)
            out["argument"].append(lemma)
            content = True
        elif tag in _ADJ_TAGS:
            out["adj"].append(lemma)
            content = True
        elif tag in _VERB_TAGS:
            if tag != "MD" and lemma != "be":
                out["verb"].append(lemma)
                content = True
        elif tag in _ADV_TAGS:
            out["adv"].append(lemma)
            if _adverb_is_content(word, lemma):
                content = True
        elif tag in _PRONOUN_TAGS:
            out["pronoun"].append(lemma)
            out["argument"].append(lemma)
        elif lemma in _DEMONSTRATIVES and tag in ("DT", "WDT"):
            if not _demonstrative_attended(triples[i + 1:n]):
                # this is a pronominal use, so it joins the pronoun (and
                # argument) classes -- same as TAACO does with its unattended
                # demonstratives
                out["pronoun"].append(lemma)
                out["argument"].append(lemma)

        if content:
            out["cw"].append(lemma)
        else:
            out["fw"].append(lemma)
    return out

cohesion_row

cohesion_row(
    document,
    *,
    mattr_window,
    connectives=(),
    keep_texts=False
)

Every lexical index for one parsed document, in column order (see header_columns, called with the same connectives; the semantic columns are computed by the caller, which holds the embedding model). Returns (token_count, values, sentence_texts, paragraph_texts) -- the texts only when keep_texts (the semantic pass needs them; pickling them back from workers is otherwise wasted weight).

Source code in src\taters\text\analyze_cohesion.py

def cohesion_row(document: List[List[List[Tuple[str, str, str]]]],
                 *, mattr_window: int,
                 connectives: Sequence[Tuple[str, List[Entry]]] = (),
                 keep_texts: bool = False,
                 ) -> Tuple[int, List[Optional[float]],
                            Optional[List[str]], Optional[List[str]]]:
    """
    Every lexical index for one parsed document, in column order (see
    `header_columns`, called with the same connectives; the semantic columns
    are computed by the caller, which holds the embedding model). Returns
    ``(token_count, values, sentence_texts, paragraph_texts)`` -- the texts
    only when ``keep_texts`` (the semantic pass needs them; pickling them
    back from workers is otherwise wasted weight).
    """
    # first, we build per-sentence and per-paragraph segments for each class.
    # note that a sentence with no members of a class still gets a slot: "no
    # adjectives here" is an absence that the adjective-overlap denominators
    # are defined over. that's how TAACO does it, and we keep it that way (and
    # say so in the guide)
    sent_segments: Dict[str, List[List[str]]] = {c: [] for c in CLASS_ORDER}
    para_segments: Dict[str, List[List[str]]] = {c: [] for c in CLASS_ORDER}
    doc_sentences_all: List[List[str]] = []
    #: (word, tag) sentences for connective matching, and the demonstrative
    #: tallies, gathered in the same pass.
    tagged_sentences: List[List[Tuple[str, str]]] = []
    dem_attended = dem_unattended = 0

    for paragraph in document:
        para_acc: Dict[str, List[str]] = {c: [] for c in CLASS_ORDER}
        for sentence in paragraph:
            classes = classify_sentence(sentence)
            doc_sentences_all.append(classes["all"])
            for c in CLASS_ORDER:
                sent_segments[c].append(classes[c])
                para_acc[c].extend(classes[c])
            counted = [(w, t) for w, _l, t in sentence if _ALNUM.search(w)]
            tagged_sentences.append(counted)
            for i, (word, _lemma, tag) in enumerate(sentence):
                if word in _DEMONSTRATIVES and tag in ("DT", "WDT"):
                    if _demonstrative_attended(sentence[i + 1:]):
                        dem_attended += 1
                    else:
                        dem_unattended += 1
        for c in CLASS_ORDER:
            para_segments[c].append(para_acc[c])

    tokens = {c: [t for seg in sent_segments[c] for t in seg]
              for c in CLASS_ORDER}
    nwords = len(tokens["all"])

    values: List[Optional[float]] = []

    # --- TTR / density family (TAACO's 15 columns, in its order) ---
    values.append(_ttr(tokens["all"]))                          # lemma_ttr
    values.append(_mattr(tokens["all"], mattr_window))          # lemma_mattr
    values.append(len(tokens["cw"]) / nwords if nwords else None)
    all_types = len(set(tokens["all"]))
    values.append(len(set(tokens["cw"])) / all_types if all_types else None)
    for cls in ("cw", "fw"):
        values.append(_ttr(tokens[cls]))                        # content/function ttr
    values.append(_mattr(tokens["fw"], mattr_window))           # function_mattr
    for cls in ("noun", "verb", "adj", "adv", "pronoun", "argument"):
        values.append(_ttr(tokens[cls]))
    values.append(_ngram_ttr(doc_sentences_all, 2))             # bigram_lemma_ttr
    values.append(_ngram_ttr(doc_sentences_all, 3))             # trigram_lemma_ttr

    # --- Lexical overlap: sentences, then paragraphs ---
    values.extend(_overlap_block(sent_segments))
    values.extend(_overlap_block(para_segments))

    # --- Synonym overlap (WordNet), sentences then paragraphs ---
    for unit_segments in (sent_segments, para_segments):
        for cls, pos in (("noun", "n"), ("verb", "v")):
            count, proportion = synonym_overlap(unit_segments[cls], pos)
            values.extend((count, proportion))

    # --- connectives: incidence per word for each category list, then the
    # --- three demonstrative columns (which come from the class machinery,
    # --- not from lists) ---
    for _name, entries in connectives:
        count = count_connectives(tagged_sentences, entries)
        values.append(count / nwords if nwords else None)
    n_dem = dem_attended + dem_unattended
    for dem in (n_dem, dem_attended, dem_unattended):
        values.append(dem / nwords if nwords else None)

    # --- Givenness ---
    n_pron = len(tokens["pronoun"])
    n_noun = len(tokens["noun"])
    values.append(n_pron / nwords if nwords else None)          # pronoun_density
    values.append(n_pron / n_noun if n_noun else None)          # pronoun_noun_ratio
    # repeated-lemma ratios. we do two things differently from TAACO here: the
    # denominator is the same class as the numerator (TAACO divided a
    # content-word count by ALL words), and we count with a Counter so that
    # it's O(n) (TAACO called list.count() per token, which is O(n^2) and gets
    # painfully slow on long documents)
    cw_counts = Counter(tokens["cw"])
    n_cw = len(tokens["cw"])
    repeated_cw = sum(1 for t in tokens["cw"] if cw_counts[t] > 1)
    values.append(repeated_cw / n_cw if n_cw else None)
    cw_pron = tokens["cw"] + tokens["pronoun"]
    cw_pron_counts = Counter(cw_pron)
    repeated_both = sum(1 for t in cw_pron if cw_pron_counts[t] > 1)
    values.append(repeated_both / len(cw_pron) if cw_pron else None)

    sent_texts = para_texts = None
    if keep_texts:
        sent_texts = []
        para_texts = []
        for paragraph in document:
            parts = []
            for sentence in paragraph:
                text = " ".join(w for w, _l, _t in sentence)
                sent_texts.append(text)
                parts.append(text)
            para_texts.append(" ".join(parts))
    return nwords, values, sent_texts, para_texts

count_connectives

count_connectives(sentences, entries)

Occurrences of a category's entries over (word, tag) sentences.

Matching is per position WITHIN a sentence -- two of TAACO's counting bugs are thereby structurally impossible: its str.count on the joined, punctuation-stripped document both undercounted adjacent repeats ("so so" counted once) and manufactured phrase matches across sentence boundaries ("...ends in. Fact is..." matched "in fact").

Source code in src\taters\text\analyze_cohesion.py

def count_connectives(sentences: Sequence[Sequence[Tuple[str, str]]],
                      entries: Sequence[Entry]) -> int:
    """
    Occurrences of a category's entries over ``(word, tag)`` sentences.

    Matching is per position WITHIN a sentence -- two of TAACO's counting
    bugs are thereby structurally impossible: its `str.count` on the joined,
    punctuation-stripped document both undercounted adjacent repeats
    ("so so" counted once) and manufactured phrase matches across sentence
    boundaries ("...ends in. Fact is..." matched "in fact").
    """
    total = 0
    singles: Dict[str, List[Optional[frozenset]]] = {}
    phrases: List[Entry] = []
    for words, constraint in entries:
        if len(words) == 1:
            singles.setdefault(words[0], []).append(constraint)
        else:
            phrases.append((words, constraint))
    for sentence in sentences:
        words = [w for w, _t in sentence]
        for i, (word, tag) in enumerate(sentence):
            for constraint in singles.get(word, ()):
                if constraint is None or tag in constraint:
                    total += 1
        for phrase, _none in phrases:
            n = len(phrase)
            for i in range(len(words) - n + 1):
                if tuple(words[i:i + n]) == phrase:
                    total += 1
    return total

header_columns

header_columns(connective_names=None, *, semantic=False)

The output columns, in order -- TAACO 2.1.3's names for every measure that survives, so results read against the TAACO literature. Connective columns are named by their list files; None means the shipped set. The four semantic columns exist only when the embedding pass runs.

Source code in src\taters\text\analyze_cohesion.py

def header_columns(connective_names: Optional[Sequence[str]] = None,
                   *, semantic: bool = False) -> List[str]:
    """The output columns, in order -- TAACO 2.1.3's names for every measure
    that survives, so results read against the TAACO literature. Connective
    columns are named by their list files; ``None`` means the shipped set.
    The four semantic columns exist only when the embedding pass runs."""
    if connective_names is None:
        connective_names = [name for name, _ in load_connective_lists()]
    cols = ["lemma_ttr", "lemma_mattr", "lexical_density_tokens",
            "lexical_density_types", "content_ttr", "function_ttr",
            "function_mattr", "noun_ttr", "verb_ttr", "adj_ttr", "adv_ttr",
            "prp_ttr", "argument_ttr", "bigram_lemma_ttr",
            "trigram_lemma_ttr"]
    for unit in ("sent", "para"):
        for cls in CLASS_ORDER:
            cols.extend((
                f"adjacent_overlap_{cls}_{unit}",
                f"adjacent_overlap_{cls}_{unit}_div_seg",
                f"adjacent_overlap_binary_{cls}_{unit}",
                f"adjacent_overlap_2_{cls}_{unit}",
                f"adjacent_overlap_2_{cls}_{unit}_div_seg",
                f"adjacent_overlap_binary_2_{cls}_{unit}",
            ))
    for unit in ("sent", "para"):
        for cls in ("noun", "verb"):
            cols.extend((f"syn_overlap_{unit}_{cls}",
                         f"syn_overlap_{unit}_{cls}_prop"))
    if semantic:
        cols.extend(("semantic_1_all_sent", "semantic_2_all_sent",
                     "semantic_1_all_para", "semantic_2_all_para"))
    cols.extend(connective_names)
    cols.extend(("all_demonstratives", "attended_demonstratives",
                 "unattended_demonstratives"))
    cols.extend(("pronoun_density", "pronoun_noun_ratio",
                 "repeated_content_lemmas",
                 "repeated_content_and_pronoun_lemmas"))
    return cols

load_connective_lists

load_connective_lists(paths=None)

Read connectives category files: one entry per line, # comments, optional TAB + TAG or TAG|TAG Penn constraint. None means the shipped categories; a directory means every .txt inside it. Each file becomes one output column named by its stem, ordered canonically (TAACO's order) with unknown stems appended alphabetically.

Source code in src\taters\text\analyze_cohesion.py

def load_connective_lists(paths: Optional[Sequence[PathLike]] = None
                          ) -> List[Tuple[str, List[Entry]]]:
    """
    Read connectives category files: one entry per line, ``#`` comments,
    optional TAB + ``TAG`` or ``TAG|TAG`` Penn constraint. ``None`` means the
    shipped categories; a directory means every ``.txt`` inside it. Each
    file becomes one output column named by its stem, ordered canonically
    (TAACO's order) with unknown stems appended alphabetically.
    """
    if paths is None:
        files = sorted(_SHIPPED_CONNECTIVES.glob("*.txt"))
    else:
        files = []
        for p in paths:
            p = Path(p)
            if p.is_dir():
                files.extend(sorted(f for f in p.rglob("*.txt") if f.is_file()))
            else:
                files.append(p)
    rank = {name: i for i, name in enumerate(_CONNECTIVE_ORDER)}
    files.sort(key=lambda f: (rank.get(f.stem, len(rank)), f.stem))

    lists: List[Tuple[str, List[Entry]]] = []
    for path in files:
        entries: List[Entry] = []
        for line in path.read_text(encoding="utf-8-sig").splitlines():
            line = line.strip()
            if not line or line.startswith("#"):
                continue
            phrase, _, tags = line.partition("\t")
            words = tuple(phrase.lower().split())
            if not words:
                continue
            constraint = frozenset(t.strip() for t in tags.split("|")
                                   if t.strip()) if tags.strip() else None
            if constraint and len(words) > 1:
                raise ValueError(
                    f"{path.name}: a tag constraint is only meaningful on a "
                    f"single word, but {' '.join(words)!r} carries one.")
            entries.append((words, constraint))
        if not entries:
            raise ValueError(f"{path} contains no connective entries.")
        lists.append((path.stem, entries))
    if not lists:
        raise ValueError("No connectives lists were found.")
    return lists

measures_guide

measures_guide()

The full text of COHESION_MEASURES.md, shipped beside this module.

Source code in src\taters\text\analyze_cohesion.py

def measures_guide() -> str:
    """The full text of COHESION_MEASURES.md, shipped beside this module."""
    return (Path(__file__).resolve().parent
            / "COHESION_MEASURES.md").read_text(encoding="utf-8")

overlap_indices

overlap_indices(segments, window)

TAACO's adjacent-overlap triple over a list of per-segment lemma lists.

For each segment i, the types of segment i are looked up in the next window segments (their concatenation -- built as a fresh list; TAACO's version appended segment i+2 into its shared input list, so the order its indices were computed in changed their values). Returns:

• proportion: total overlapping types / total types of the source segments (TAACO's adjacent_overlap_X: type-normalized, not word-normalized, despite the name -- kept, and documented).
• per-pair mean: total overlapping types / number of comparisons (_div_seg; unbounded above).
• binary: share of comparisons with at least one overlapping type.

A document with too few segments returns (None, None, None) -- NA, where TAACO wrote 0.0 and made "one paragraph" look like "zero cohesion".

Source code in src\taters\text\analyze_cohesion.py

def overlap_indices(segments: Sequence[Sequence[str]],
                    window: int) -> Tuple[Optional[float], Optional[float],
                                          Optional[float]]:
    """
    TAACO's adjacent-overlap triple over a list of per-segment lemma lists.

    For each segment ``i``, the *types* of segment ``i`` are looked up in the
    next ``window`` segments (their concatenation -- built as a fresh list;
    TAACO's version appended segment i+2 into its shared input list, so the
    order its indices were computed in changed their values). Returns:

    * proportion: total overlapping types / total types of the source
      segments (TAACO's ``adjacent_overlap_X``: type-normalized, not
      word-normalized, despite the name -- kept, and documented).
    * per-pair mean: total overlapping types / number of comparisons
      (``_div_seg``; unbounded above).
    * binary: share of comparisons with at least one overlapping type.

    A document with too few segments returns ``(None, None, None)`` -- NA,
    where TAACO wrote 0.0 and made "one paragraph" look like "zero
    cohesion".
    """
    n = len(segments)
    comparisons = n - window
    if comparisons < 1:
        return None, None, None
    overlap_total = 0
    type_total = 0
    hits = 0
    for i in range(comparisons):
        source_types = set(segments[i])
        target = set()
        for j in range(1, window + 1):
            target.update(segments[i + j])
        overlapping = len(source_types & target)
        overlap_total += overlapping
        type_total += len(source_types)
        if overlapping:
            hits += 1
    proportion = overlap_total / type_total if type_total else 0.0
    return proportion, overlap_total / comparisons, hits / comparisons

synonym_overlap

synonym_overlap(segments, pos)

Adjacent-segment synonym overlap; returns (count, proportion).

count is TAACO's definition, kept for comparability and documented as what it is (audit finding 4.15): for each type of segment i, one hit per word of segment i+1 whose synonym set contains it -- a multiply- counting, unbounded count, averaged over segment pairs, NOT comparable across texts with different sentence lengths. proportion is the normalized companion this implementation adds: source types matched by at least one next-segment word, over total source types -- the same scale as the lexical overlap indices. Too few segments: (None, None).

Source code in src\taters\text\analyze_cohesion.py

def synonym_overlap(segments: Sequence[Sequence[str]],
                    pos: str) -> Tuple[Optional[float], Optional[float]]:
    """
    Adjacent-segment synonym overlap; returns ``(count, proportion)``.

    ``count`` is TAACO's definition, kept for comparability and documented
    as what it is (audit finding 4.15): for each *type* of segment i, one
    hit per word of segment i+1 whose synonym set contains it -- a multiply-
    counting, unbounded count, averaged over segment pairs, NOT comparable
    across texts with different sentence lengths. ``proportion`` is the
    normalized companion this implementation adds: source types matched by
    at least one next-segment word, over total source types -- the same
    scale as the lexical overlap indices. Too few segments: ``(None, None)``.
    """
    n = len(segments)
    if n < 2:
        return None, None
    count_total = 0
    matched_types = 0
    type_total = 0
    for i in range(n - 1):
        source_types = set(segments[i])
        next_synonyms = [_synonyms(w, pos) for w in segments[i + 1]]
        type_total += len(source_types)
        for t in source_types:
            hits = sum(1 for s in next_synonyms if t in s)
            count_total += hits
            if hits:
                matched_types += 1
    proportion = matched_types / type_total if type_total else 0.0
    return count_total / (n - 1), proportion

taters.text.topic_model_mem

Topic model: the Meaning Extraction Method (MEM).

Chung & Pennebaker's MEM (2008): take a document-term matrix over frequent content terms, run a PCA with varimax rotation, and read the rotated components as themes -- clusters of words that rise and fall together across documents. Each document gets a score per theme; each term gets a loading per theme.

Two commitments shape this module:

• Exact and out-of-core, without heavy dependencies. A MEM matrix is long (documents) but narrow (hundreds to a few thousand terms), so the fit streams the rows once to accumulate the column means and the p-by-p cross-product matrix, then eigendecomposes the correlation matrix in memory. That is the exact full PCA -- no randomized SVD, no seed, no dask -- with memory bounded by the vocabulary width, never the corpus length. Should a vocabulary ever be too wide for p-by-p to fit, that is the moment to reach for an out-of-core SVD, behind this same interface.

• A fitted model is a reusable instrument. topic_model_mem writes a model file carrying everything needed to score new text on the same themes: the exact vocabulary (tagged keys included), the tokenizer/engine settings that produced it, the weighting, the training means and standard deviations, and the projection. apply_mem_model rebuilds the same document-term matrix for new texts -- through the very same scan and weighting code the DTM step uses, imported rather than copied -- and projects it. Applying a model to its own training texts reproduces the training scores exactly.

Eigenvalues here are those of the correlation matrix (they average 1.0), which is what the retention rules are stated over: n_components=0 picks the count by parallel analysis (a theme is kept while its eigenvalue beats what a random matrix of the same shape gives at that rank) or, on request, by the Kaiser criterion (keep eigenvalue >= a cutoff).

Two files come out of this, because there are two quantities and only one of them is an eigenvalue. The eigenvalues file is the spectrum the rule read, every rank beside what chance reaches there. The theme variance file is each finished theme's sum of squared loadings -- which is the same quantity before rotation, and deliberately redistributed by it, so it comes out much flatter than the spectrum. Classic MEM tooling reports the second; this reports both, apart, because they are sorted lists of different things and a shared table made that look like a pairing.

apply_mem_model

apply_mem_model(
    *,
    model_json,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    on_progress=None,
    out_features_csv=None,
    overwrite_existing=False,
    workers=0,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    device="auto",
    rounding=4
)

Score new texts on the themes of a saved MEM model.

The model file (from :func:topic_model_mem) carries the vocabulary, tokenizer settings, weighting, and projection of the original fit; this function rebuilds the same document-term matrix for the new texts -- through the same scan and weighting code the matrix step uses -- then standardizes with the training means and deviations and projects. Applying a model to its own training texts reproduces the training scores exactly.

Parameters:

Name	Type	Description	Default
model_json	PathLike	A *_model.json written by :func:topic_model_mem. A folder (or a list, as the library hands over) is accepted when it resolves to exactly one model file; anything else refuses and says how to pick.	required
csv_path	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
txt_dir	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
analysis_csv	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
gathered_csv	Optional[PathLike]	The same input contract as the other text analyzers: a spreadsheet of texts, a folder of documents, or a prebuilt analysis-ready CSV.	None
out_features_csv	str or Path	Output file path. If None, defaults to ./features/topic_model_mem/<analysis_ready_filename>.	None
overwrite_existing	bool	If False and the output file already exists, skip processing and return the path.	False
workers	int	Parallel processes for reading documents during the gather. 0 means automatic: three-quarters of the logical cores.	0
encoding	str	Encoding for reading and writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	When gathering from a CSV, name(s) of the column(s) containing text.	("text",)
id_cols	Sequence[str] or None	Optional ID columns that identify each row when gathering from CSV.	None
mode	('concat', 'separate')	Gathering behavior when multiple text columns are provided.	"concat"
group_by	Sequence[str] or None	Optional grouping keys used during CSV gathering.	None
delimiter	str	Column separator of the input CSV.	","
pattern	str	Which files to read when gathering from a folder of documents.	every document type
device	('auto', 'cuda', 'cpu')	Where Stanza runs, if the model was built with the stanza engine -- a runtime choice, deliberately not stored in the model.	"auto"
rounding	int	Decimal places for the theme scores.	4

Returns:

Type	Description
Path	out_features_csv: text_id, token_count, then one column per theme of the model.

Source code in src\taters\text\topic_model_mem.py

@records_settings(
    # the model file is the whole instrument: vocabulary, loadings, and the
    # tokenizer settings the matrix was built with. we compare it by content
    # so that a table scored with a same-named model with different themes
    # gets told apart from one scored with the model a ridge was fitted beside
    binding=TEXT_INPUT, grain=TEXT_GRAIN, assets={"model_json": None},
    outputs=("out_features_csv",), bookkeeping=("token_count",))
def apply_mem_model(
    *,
    model_json: PathLike,

    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    on_progress: Optional[Callable[[int, int], None]] = None,

    # ----- Output -----
    out_features_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    workers: int = 0,

    # ====== SHARED I/O OPTIONS ======
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS (used when csv_path is provided) ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ====== TXT FOLDER GATHER OPTIONS (used when txt_dir is provided) ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ====== APPLY OPTIONS ======
    device: str = "auto",
    rounding: int = 4,
) -> Path:
    """
    Score new texts on the themes of a saved MEM model.

    The model file (from :func:`topic_model_mem`) carries the vocabulary,
    tokenizer settings, weighting, and projection of the original fit; this
    function rebuilds the same document-term matrix for the new texts --
    through the same scan and weighting code the matrix step uses -- then
    standardizes with the *training* means and deviations and projects.
    Applying a model to its own training texts reproduces the training
    scores exactly.

    Parameters
    ----------
    model_json
        A ``*_model.json`` written by :func:`topic_model_mem`. A folder (or a
        list, as the library hands over) is accepted when it resolves to
        exactly one model file; anything else refuses and says how to pick.
    csv_path, txt_dir, analysis_csv, gathered_csv
        The same input contract as the other text analyzers: a spreadsheet of
        texts, a folder of documents, or a prebuilt analysis-ready CSV.
    out_features_csv : str or pathlib.Path, optional
        Output file path. If ``None``, defaults to
        ``./features/topic_model_mem/<analysis_ready_filename>``.
    overwrite_existing : bool, default=False
        If ``False`` and the output file already exists, skip processing and
        return the path.
    workers : int, default=0
        Parallel processes for reading documents during the gather. ``0``
        means automatic: three-quarters of the logical cores.
    encoding : str, default="utf-8-sig"
        Encoding for reading and writing CSV files.
    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV, name(s) of the column(s) containing text.
    id_cols : Sequence[str] or None, optional
        Optional ID columns that identify each row when gathering from CSV.
    mode : {"concat", "separate"}, default="concat"
        Gathering behavior when multiple text columns are provided.
    group_by : Sequence[str] or None, optional
        Optional grouping keys used during CSV gathering.
    delimiter : str, default=","
        Column separator of the *input* CSV.
    pattern : str, default=every document type
        Which files to read when gathering from a folder of documents.
    device : {"auto", "cuda", "cpu"}, default="auto"
        Where Stanza runs, if the model was built with the stanza engine --
        a runtime choice, deliberately not stored in the model.
    rounding : int, default=4
        Decimal places for the theme scores.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id``, ``token_count``, then one column
        per theme of the model.
    """
    import numpy as np

    model = _load_model(one_model_path(model_json))
    text_cfg = model["text"]
    matrix_cfg = model["matrix"]
    fit = model["model"]

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        # `_applied`, so that fitting and applying with the defaults do not
        # write to one file -- and so their matrix folders, derived from this
        # stem, can never be the same one.
        out_features_csv = (Path.cwd() / "features" / "topic_model_mem"
                            / f"{analysis_ready.stem}_applied{analysis_ready.suffix}")
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)
    if not overwrite_existing and out_features_csv.is_file():
        print("MEM theme scores output file already exists; returning existing file.")
        return out_features_csv

    # 2) now we rebuild the instrument from the model, exactly as it was trained
    if text_cfg["engine"] == "stanza":
        announce(on_progress, "loading the stanza pipeline (first use "
                              "downloads its model)")
    # a model saved before this setting existed was fitted on a vocabulary
    # that still had its punctuation in it, so a missing key means "keep it"
    keep_punctuation = bool(text_cfg.get("keep_punctuation", True))
    stream = make_token_stream(
        text_cfg["lemmatize"], text_cfg["pos_tagged"],
        engine=text_cfg["engine"], tokenizer=text_cfg["tokenizer"],
        stanza_lang=text_cfg["stanza_lang"], device=device,
        keep_punctuation=keep_punctuation)
    terms: List[str] = list(matrix_cfg["terms"])
    term_index = {g: i for i, g in enumerate(terms)}
    max_n = max(g.count(" ") + 1 for g in terms)
    idf: List[float] = list(matrix_cfg["idf"])
    weighting = matrix_cfg["weighting"]
    matrix_rounding = int(matrix_cfg["rounding"])
    kept = np.asarray(fit["kept"], dtype=int)
    mu = np.asarray(fit["mu"], dtype=np.float64)
    sigma = np.asarray(fit["sigma"], dtype=np.float64)
    projection = np.asarray(fit["projection"], dtype=np.float64)
    theme_names = list(fit["themes"])

    # 3) scan and weight on the shared pooled-row driver. this is the same
    #    worker the DTM step runs, so new text gets scored through the exact
    #    same code. then we standardize with the TRAINING statistics and
    #    project. stanza-backed models stay single-process (see
    #    `pooled_text_workers`)
    from ..helpers.row_map import map_text_rows
    from .build_doc_term_matrix import _cells_of, _init_scan_worker, _scan_in_worker
    from .ngram_prep import pooled_text_workers

    stream_args = dict(lemmatize=text_cfg["lemmatize"],
                       pos_tagged=text_cfg["pos_tagged"],
                       engine=text_cfg["engine"],
                       tokenizer=text_cfg["tokenizer"],
                       stanza_lang=text_cfg["stanza_lang"], device=device,
                       keep_punctuation=keep_punctuation)
    with atomic_write(out_features_csv, newline="", encoding=encoding) as out:
        writer = csv.writer(out)
        writer.writerow(["text_id", "token_count", *theme_names])
        for row, (n_tokens, cells) in map_text_rows(
                analysis_ready, encoding=encoding,
                workers=lambda n_rows: pooled_text_workers(
                    workers, n_rows, engine=text_cfg["engine"],
                    tokenizer=text_cfg["tokenizer"],
                    lemmatize=text_cfg["lemmatize"],
                    pos_tagged=text_cfg["pos_tagged"]),
                message="scoring themes", on_progress=on_progress,
                inline_fn=lambda pair: _cells_of(
                    stream, term_index, max_n, weighting, idf,
                    matrix_rounding, pair[1]),
                pool_fn=_scan_in_worker,
                initializer=_init_scan_worker,
                initargs=(stream_args, terms, weighting, idf, matrix_rounding)):
            writer.writerow(_theme_row(row.get("text_id", ""), n_tokens,
                                       cells, kept, mu, sigma, projection,
                                       rounding))

    return out_features_csv

topic_model_mem

topic_model_mem(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    workers=0,
    device="auto",
    out_features_csv=None,
    out_model_json=None,
    out_loadings_csv=None,
    out_eigenvalues_csv=None,
    out_theme_variance_csv=None,
    overwrite_existing=False,
    on_progress=None,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    ngram_n=1,
    stoplist_paths=None,
    min_freq=5,
    min_obs_pct=0.1,
    min_token_count=10,
    min_npmi=None,
    lemmatize=False,
    pos_tagged=False,
    engine="nltk",
    tokenizer="potts",
    stanza_lang="en",
    keep_punctuation=False,
    weighting="count",
    matrix_rounding=4,
    vocab_min_freq=0,
    vocab_min_obs_pct=0,
    vocab_rule="top_n",
    vocab_top_n=500,
    vocab_rank_by="obs_pct",
    n_components=0,
    k_selection="parallel",
    kaiser_cutoff=1.5,
    k_values=_topics.DEFAULT_K_VALUES,
    coherence_metric="npmi",
    top_terms=15,
    rotation=True,
    rounding=4
)

Fit MEM themes to a corpus; write scores, the matrix, and a reusable model.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse. MEM builds its own frequency list and document-term matrix from it, into a <results-stem>_matrix folder beside the results -- every topic model needs a different matrix, so sharing one meant two of them quietly rebuilding over each other.	None
txt_dir	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse. MEM builds its own frequency list and document-term matrix from it, into a <results-stem>_matrix folder beside the results -- every topic model needs a different matrix, so sharing one meant two of them quietly rebuilding over each other.	None
analysis_csv	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse. MEM builds its own frequency list and document-term matrix from it, into a <results-stem>_matrix folder beside the results -- every topic model needs a different matrix, so sharing one meant two of them quietly rebuilding over each other.	None
gathered_csv	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse. MEM builds its own frequency list and document-term matrix from it, into a <results-stem>_matrix folder beside the results -- every topic model needs a different matrix, so sharing one meant two of them quietly rebuilding over each other.	None
workers	int	Worker processes for gathering, counting and scoring. 0 picks a sensible number for the machine and the size of the job.	0
device	('auto', 'cuda', 'cpu')	Where Stanza runs, when engine="stanza". The NLTK engine is CPU-only either way, and the fit itself is linear algebra on the CPU.	"auto"
text_cols	Sequence[str]	When gathering from a CSV, name(s) of the column(s) containing text.	("text",)
id_cols	Sequence[str] or None	Optional ID columns that identify each row when gathering from CSV.	None
mode	('concat', 'separate')	Gathering behavior when multiple text columns are provided: "concat" joins them into one text per row; "separate" measures each column on its own.	"concat"
group_by	Sequence[str] or None	Optional grouping keys used during CSV gathering (e.g. ["speaker"]) -- one document per group instead of one per row.	None
pattern	str	Which files to read when gathering from a folder of documents (globs, ;-separated). Only used with txt_dir.	every document type
ngram_n	int	Highest n-gram order to consider for the vocabulary. Themes are usually built from single words; raise it to let phrases compete.	1
stoplist_paths	Sequence[str or Path] or None	Word lists to drop before counting. Function words carry grammar rather than topic, and leaving them in gives a first theme that is mostly "the".	None
min_freq	int	The first of two cuts: a term used fewer times than this anywhere in the corpus is never counted, so it cannot reach the frequency list. The vocab_* settings below then decide how many of the survivors the model sees -- which is what vocab_min_freq does, and why the two are not the same setting.	5
min_obs_pct	float	Also the first cut: drop terms found in fewer than this percent of documents. The setting that matters most for themes -- a term almost nobody uses cannot covary with anything.	0.10
min_token_count	int	Skip whole documents shorter than this many tokens. A filter on texts, not on terms, despite sitting among the term filters.	10
min_npmi	float	Optional collocation threshold for orders above one. Default None: the metric is reported and filtering stays an analysis decision.	None
out_features_csv	str or Path	Per-document theme scores. Defaults to ./features/topic_model_mem/<gathered filename>. The model, loadings, and eigenvalue files are written next to it unless given their own paths (<name>_model.json, <name>_loadings.csv, <name>_eigenvalues.csv).	None
out_model_json	optional	The reusable model (see :func:apply_mem_model) and the term-by-theme rotated loadings.	None
out_loadings_csv	optional	The reusable model (see :func:apply_mem_model) and the term-by-theme rotated loadings.	None
out_eigenvalues_csv	optional	The spectrum: one row per rank with the correlation matrix's eigenvalue, the level chance alone reaches at that rank (under parallel analysis), and whether it was kept. This is what the retention rule read and the curve to judge signal by. Every rank is listed, not just the kept ones, so you can see where the curve crosses.	None
out_theme_variance_csv	optional	What each finished theme accounts for: its sum of squared loadings and that as a percent. Deliberately a separate file from the eigenvalues, because the two do not line up. Varimax rotates the kept axes within the space they span, so a rotated theme is a remix of all of them and theme 3 is not built from eigenvector 3. Both lists come out sorted descending, which is the only thing they share -- and putting them in one table made that coincidence look like a correspondence.	None
overwrite_existing	bool	If False and the scores file already exists, skip and return it.	False
encoding	str	Encoding for reading and writing CSV files.	"utf-8-sig"
lemmatize	bool	Reduce words to a dictionary form before counting, so "running", "runs" and "ran" become one term instead of three.	False
pos_tagged	bool	Keep each word's part of speech attached, so "book" the noun and "book" the verb count as different terms.	False
engine	('nltk', 'stanza')	Which toolkit does the tagging and lemmatizing. NLTK is fast and English-only; Stanza handles many languages and is much slower, and downloads a model the first time you use it.	"nltk"
tokenizer	('potts', 'stanza')	Which rules split the text into words. "potts" keeps emoticons and hashtags intact, which is usually what you want for social media; "stanza" uses Stanza's own splitter.	"potts"
stanza_lang	str	The language code for Stanza, when the engine is Stanza.	"en"
keep_punctuation	bool	Count punctuation marks as terms of their own.	False
weighting	('count', 'binary', 'relfreq', 'tfidf')	The matrix's cell weighting, recorded so apply weights new text the same way.	"count"
matrix_rounding	int	The rounding the matrix step used (its default). Only affects relfreq/tfidf cells; recorded so apply reproduces them digit for digit.	4
vocab_rule	('top_n', 'min_obs_pct', 'min_freq')	Which of the surviving terms become columns of the matrix. This is the second of two cuts and the source of a long-standing confusion. min_freq and min_obs_pct above decide which terms get counted at all, so they never reach the frequency list; these decide how many of the survivors the model actually sees. The names are nearly the same and the jobs are not. top_n takes the strongest vocab_top_n terms by vocab_rank_by. The other two rules keep everything above a threshold instead, and read vocab_min_freq or vocab_min_obs_pct respectively -- whichever one does not match the rule is ignored.	"top_n"
vocab_top_n	int	How many terms the model gets under vocab_rule="top_n".	500
vocab_rank_by	('obs_pct', 'frequency')	What "strongest" means when taking the top N. obs_pct ranks by how many documents a term appears in; frequency by how often it appears in total -- so a word one document shouts four hundred times tops the frequency ranking and all but vanishes under obs_pct. Spread is what themes are made of, which is why this is the default.	"obs_pct"
vocab_min_freq	float	The threshold under vocab_rule="min_freq": keep terms used at least this many times across the corpus.	0
vocab_min_obs_pct	float	The threshold under vocab_rule="min_obs_pct": keep terms found in at least this percent of documents.	0
n_components	int	How many themes to keep. 0 picks automatically by k_selection; an explicit number is honored up to the number of non-constant terms.	0
k_selection	('parallel', 'kaiser', 'coherence', 'coherence_exclusivity')	How 0 chooses. parallel is parallel analysis: a component is kept while its eigenvalue beats the 95th percentile of what fifty random data sets of the same shape produce at the same rank. That last part is why it is the default. A document-term matrix is wide, and on a wide matrix chance alone makes large eigenvalues: at 938 documents by 515 terms, random noise produces eigenvalues up to about 3.0, so any fixed cutoff below that keeps themes indistinguishable from noise. Parallel analysis works out that ceiling for your matrix instead of assuming one. kaiser keeps every component whose eigenvalue beats kaiser_cutoff, which is simple and reproducible but cannot know the shape of your data. Both are cheap -- they read eigenvalues the fit has already computed. coherence and coherence_exclusivity instead fit at each of k_values and score the themes' words, which is the same question LDA and NMF answer and lets the three be compared on one footing. They cost a varimax rotation per candidate count, not a whole refit, because the eigen-decomposition underneath is computed once.	"parallel"
kaiser_cutoff	float	The eigenvalue a theme has to beat under kaiser. The textbook value is 1.0 -- one term's worth of variance -- but on a vocabulary of hundreds of terms that keeps almost everything (one real corpus gave 101 themes), so this starts higher. Raising it further is often right: a cutoff only means something if it clears the level chance alone reaches at your matrix's shape, which is roughly (1 + sqrt(terms / documents)) ** 2 and is what parallel measures rather than guesses.	1.5
k_values	str or sequence of int	The theme counts the two scoring rules try. "5,10,20" as well as a list. Counts this corpus is too small to support are skipped and named rather than ending the run.	DEFAULT_K_VALUES
coherence_metric	('npmi', 'umass')	Which coherence. NPMI is bounded, which is what lets it be balanced against exclusivity without rescaling.	"npmi"
top_terms	int	How many of a theme's words the scoring rules look at. Only its positive pole counts: a theme is bipolar and its two ends anti-correlate by construction, so mixing them would make every theme score badly and the comparison across counts meaningless.	15
rotation	bool	Varimax-rotate the components. MEM is defined over rotated loadings; turn off only to inspect the raw principal axes.	True
rounding	int	Decimal places for theme scores, loadings, and eigenvalues.	4

Returns:

Type	Description
Path	out_features_csv: text_id, token_count, then Theme_1..Theme_k.

Source code in src\taters\text\topic_model_mem.py

@records_settings(
    # the corpus is our binding now. we used to bind the matrix and the
    # frequency list, because somebody else built them and the chain had to
    # walk back through *their* records to reach the gather. this step builds
    # them itself, so the gather is one hop away and the shared text-input
    # declaration is the right one -- the same one every other text step uses.
    binding=TEXT_INPUT, grain=TEXT_GRAIN,
    outputs=("out_features_csv", "out_model_json", "out_loadings_csv",
             "out_eigenvalues_csv", "out_theme_variance_csv"),
    # the themes are fitted to this corpus, so the honest way to measure them
    # on another is to apply the saved model. when we refit instead, a second
    # study picked a different number of themes, and a ridge fitted on the
    # first one couldn't be scored at all
    replay=(f"{__name__}:apply_mem_model", {"model_json": "out_model_json"}),
    bookkeeping=("token_count",))
def topic_model_mem(
    *,
    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    workers: int = 0,
    device: str = "auto",
    out_features_csv: Optional[PathLike] = None,
    out_model_json: Optional[PathLike] = None,
    out_loadings_csv: Optional[PathLike] = None,
    out_eigenvalues_csv: Optional[PathLike] = None,
    out_theme_variance_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    on_progress: Optional[Callable[[int, int], None]] = None,
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ====== TXT FOLDER GATHER OPTIONS ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ----- how the vocabulary is chosen (the frequency list this builds) -----
    ngram_n: int = 1,
    stoplist_paths: Optional[Sequence[PathLike]] = None,
    min_freq: int = 5,
    min_obs_pct: float = 0.10,
    min_token_count: int = 10,
    min_npmi: Optional[float] = None,

    # ----- how the matrix is made. this step builds it, and records every one
    # ----- of these into the model so `apply_mem_model` can rebuild it exactly
    lemmatize: bool = False,
    pos_tagged: bool = False,
    engine: Literal["nltk", "stanza"] = "nltk",
    tokenizer: Literal["potts", "stanza"] = "potts",
    stanza_lang: str = "en",
    keep_punctuation: bool = False,
    weighting: Literal["count", "binary", "relfreq", "tfidf"] = "count",
    matrix_rounding: int = 4,
    vocab_min_freq: float = 0,
    vocab_min_obs_pct: float = 0,
    vocab_rule: Literal["top_n", "min_obs_pct", "min_freq"] = "top_n",
    vocab_top_n: int = 500,
    vocab_rank_by: Literal["obs_pct", "frequency"] = "obs_pct",

    # ----- MEM options -----
    n_components: int = 0,
    k_selection: Literal["parallel", "kaiser", "coherence",
                         "coherence_exclusivity"] = "parallel",
    kaiser_cutoff: float = 1.5,
    k_values: Union[str, Sequence[int]] = _topics.DEFAULT_K_VALUES,
    coherence_metric: Literal["npmi", "umass"] = "npmi",
    top_terms: int = 15,
    rotation: bool = True,
    rounding: int = 4,
) -> Path:
    """
    Fit MEM themes to a corpus; write scores, the matrix, and a reusable model.

    Parameters
    ----------
    csv_path, txt_dir, analysis_csv, gathered_csv
        The corpus, given exactly one of these ways: a spreadsheet, a folder of
        documents, an already-gathered analysis-ready table, or a gathered
        table to write and reuse. MEM builds its own frequency list and
        document-term matrix from it, into a ``<results-stem>_matrix`` folder
        beside the results -- every topic model needs a different matrix, so
        sharing one meant two of them quietly rebuilding over each other.
    workers : int, default=0
        Worker processes for gathering, counting and scoring. 0 picks a
        sensible number for the machine and the size of the job.
    device : {"auto", "cuda", "cpu"}, default="auto"
        Where Stanza runs, when ``engine="stanza"``. The NLTK engine is
        CPU-only either way, and the fit itself is linear algebra on the CPU.
    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV, name(s) of the column(s) containing text.
    id_cols : Sequence[str] or None, optional
        Optional ID columns that identify each row when gathering from CSV.
    mode : {"concat", "separate"}, default="concat"
        Gathering behavior when multiple text columns are provided:
        ``"concat"`` joins them into one text per row; ``"separate"`` measures
        each column on its own.
    group_by : Sequence[str] or None, optional
        Optional grouping keys used during CSV gathering (e.g. ``["speaker"]``)
        -- one document per group instead of one per row.
    pattern : str, default=every document type
        Which files to read when gathering from a folder of documents
        (globs, ``;``-separated). Only used with ``txt_dir``.
    ngram_n : int, default=1
        Highest n-gram order to consider for the vocabulary. Themes are
        usually built from single words; raise it to let phrases compete.
    stoplist_paths : Sequence[str or pathlib.Path] or None, optional
        Word lists to drop before counting. Function words carry grammar
        rather than topic, and leaving them in gives a first theme that is
        mostly "the".
    min_freq : int, default=5
        The **first** of two cuts: a term used fewer times than this anywhere
        in the corpus is never counted, so it cannot reach the frequency list.
        The ``vocab_*`` settings below then decide how many of the survivors
        the model sees -- which is what ``vocab_min_freq`` does, and why the
        two are not the same setting.
    min_obs_pct : float, default=0.10
        Also the first cut: drop terms found in fewer than this *percent* of
        documents. The setting that matters most for themes -- a term almost
        nobody uses cannot covary with anything.
    min_token_count : int, default=10
        Skip whole *documents* shorter than this many tokens. A filter on
        texts, not on terms, despite sitting among the term filters.
    min_npmi : float, optional
        Optional collocation threshold for orders above one. Default None:
        the metric is reported and filtering stays an analysis decision.
    out_features_csv : str or pathlib.Path, optional
        Per-document theme scores. Defaults to
        ``./features/topic_model_mem/<gathered filename>``. The model, loadings,
        and eigenvalue files are written next to it unless given their own
        paths (``<name>_model.json``, ``<name>_loadings.csv``,
        ``<name>_eigenvalues.csv``).
    out_model_json, out_loadings_csv : optional
        The reusable model (see :func:`apply_mem_model`) and the term-by-theme
        rotated loadings.
    out_eigenvalues_csv : optional
        The **spectrum**: one row per rank with the correlation matrix's
        eigenvalue, the level chance alone reaches at that rank (under
        parallel analysis), and whether it was kept. This is what the
        retention rule read and the curve to judge signal by. Every rank is
        listed, not just the kept ones, so you can see where the curve crosses.
    out_theme_variance_csv : optional
        What each finished **theme** accounts for: its sum of squared loadings
        and that as a percent.

        Deliberately a separate file from the eigenvalues, because the two do
        not line up. Varimax rotates the kept axes within the space they span,
        so a rotated theme is a remix of all of them and theme 3 is not built
        from eigenvector 3. Both lists come out sorted descending, which is
        the only thing they share -- and putting them in one table made that
        coincidence look like a correspondence.
    overwrite_existing : bool, default=False
        If ``False`` and the scores file already exists, skip and return it.
    encoding : str, default="utf-8-sig"
        Encoding for reading and writing CSV files.
    lemmatize : bool, default=False
        Reduce words to a dictionary form before counting, so "running",
        "runs" and "ran" become one term instead of three.
    pos_tagged : bool, default=False
        Keep each word's part of speech attached, so "book" the noun and
        "book" the verb count as different terms.
    engine : {"nltk", "stanza"}, default="nltk"
        Which toolkit does the tagging and lemmatizing. NLTK is fast and
        English-only; Stanza handles many languages and is much slower, and
        downloads a model the first time you use it.
    tokenizer : {"potts", "stanza"}, default="potts"
        Which rules split the text into words. "potts" keeps emoticons and
        hashtags intact, which is usually what you want for social media;
        "stanza" uses Stanza's own splitter.
    stanza_lang : str, default="en"
        The language code for Stanza, when the engine is Stanza.
    keep_punctuation : bool, default=False
        Count punctuation marks as terms of their own.
    weighting : {"count", "binary", "relfreq", "tfidf"}, default="count"
        The matrix's cell weighting, recorded so apply weights new text the
        same way.
    matrix_rounding : int, default=4
        The ``rounding`` the matrix step used (its default). Only affects
        relfreq/tfidf cells; recorded so apply reproduces them digit for
        digit.
    vocab_rule : {"top_n", "min_obs_pct", "min_freq"}, default="top_n"
        Which of the surviving terms become columns of the matrix.

        This is the **second** of two cuts and the source of a long-standing
        confusion. ``min_freq`` and ``min_obs_pct`` above decide which terms
        get counted at all, so they never reach the frequency list; these
        decide how many of the survivors the model actually sees. The names
        are nearly the same and the jobs are not.

        ``top_n`` takes the strongest ``vocab_top_n`` terms by
        ``vocab_rank_by``. The other two rules keep everything above a
        threshold instead, and read ``vocab_min_freq`` or
        ``vocab_min_obs_pct`` respectively -- whichever one does not match the
        rule is ignored.
    vocab_top_n : int, default=500
        How many terms the model gets under ``vocab_rule="top_n"``.
    vocab_rank_by : {"obs_pct", "frequency"}, default="obs_pct"
        What "strongest" means when taking the top N. ``obs_pct`` ranks by how
        many documents a term appears in; ``frequency`` by how often it
        appears in total -- so a word one document shouts four hundred times
        tops the frequency ranking and all but vanishes under ``obs_pct``.
        Spread is what themes are made of, which is why this is the default.
    vocab_min_freq : float, default=0
        The threshold under ``vocab_rule="min_freq"``: keep terms used at
        least this many times across the corpus.
    vocab_min_obs_pct : float, default=0
        The threshold under ``vocab_rule="min_obs_pct"``: keep terms found in
        at least this percent of documents.
    n_components : int, default=0
        How many themes to keep. ``0`` picks automatically by
        ``k_selection``; an explicit number is honored up to the number of
        non-constant terms.
    k_selection : {"parallel", "kaiser", "coherence", "coherence_exclusivity"}, default="parallel"
        How ``0`` chooses.

        ``parallel`` is parallel analysis: a component is kept while its
        eigenvalue beats the 95th percentile of what fifty random data sets of
        the *same shape* produce at the same rank. That last part is why it is
        the default. A document-term matrix is wide, and on a wide matrix
        chance alone makes large eigenvalues: at 938 documents by 515 terms,
        random noise produces eigenvalues up to about 3.0, so any fixed cutoff
        below that keeps themes indistinguishable from noise. Parallel analysis
        works out that ceiling for your matrix instead of assuming one.

        ``kaiser`` keeps every component whose eigenvalue beats
        ``kaiser_cutoff``, which is simple and reproducible but cannot know
        the shape of your data. Both are cheap -- they read eigenvalues the
        fit has already computed.

        ``coherence`` and ``coherence_exclusivity`` instead *fit* at each of
        ``k_values`` and score the themes' words, which is the same question
        LDA and NMF answer and lets the three be compared on one footing.
        They cost a varimax rotation per candidate count, not a whole refit,
        because the eigen-decomposition underneath is computed once.
    kaiser_cutoff : float, default=1.5
        The eigenvalue a theme has to beat under ``kaiser``. The textbook
        value is 1.0 -- one term's worth of variance -- but on a vocabulary of
        hundreds of terms that keeps almost everything (one real corpus gave
        101 themes), so this starts higher. Raising it further is often right:
        a cutoff only means something if it clears the level chance alone
        reaches at your matrix's shape, which is roughly
        ``(1 + sqrt(terms / documents)) ** 2`` and is what ``parallel``
        measures rather than guesses.
    k_values : str or sequence of int
        The theme counts the two scoring rules try. ``"5,10,20"`` as well as
        a list. Counts this corpus is too small to support are skipped and
        named rather than ending the run.
    coherence_metric : {"npmi", "umass"}, default="npmi"
        Which coherence. NPMI is bounded, which is what lets it be balanced
        against exclusivity without rescaling.
    top_terms : int, default=15
        How many of a theme's words the scoring rules look at. Only its
        *positive* pole counts: a theme is bipolar and its two ends
        anti-correlate by construction, so mixing them would make every theme
        score badly and the comparison across counts meaningless.
    rotation : bool, default=True
        Varimax-rotate the components. MEM is defined over rotated loadings;
        turn off only to inspect the raw principal axes.
    rounding : int, default=4
        Decimal places for theme scores, loadings, and eigenvalues.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id``, ``token_count``, then
        ``Theme_1..Theme_k``.
    """
    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        out_features_csv = (Path.cwd() / "features" / "topic_model_mem"
                            / analysis_ready.name)
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)
    stem = out_features_csv.stem
    model_path = Path(out_model_json) if out_model_json else \
        out_features_csv.with_name(f"{stem}_model.json")
    loadings_path = Path(out_loadings_csv) if out_loadings_csv else \
        out_features_csv.with_name(f"{stem}_loadings.csv")
    eigen_path = Path(out_eigenvalues_csv) if out_eigenvalues_csv else \
        out_features_csv.with_name(f"{stem}_eigenvalues.csv")
    variance_path = Path(out_theme_variance_csv) if out_theme_variance_csv else \
        out_features_csv.with_name(f"{stem}_theme_variance.csv")

    if not overwrite_existing and out_features_csv.is_file():
        print("MEM theme scores output file already exists; returning existing file.")
        return out_features_csv

    # 1) build this model's own frequency list and matrix.
    #
    # these used to arrive as two paths, produced by separate steps that a
    # pipeline wired in -- and the vocabulary was re-derived here and held up
    # against the matrix's header, because nothing guaranteed the two had been
    # made with the same settings. that check is gone along with the problem it
    # was catching: we make both, so they cannot disagree.
    #
    # the reason to make them rather than share them is that every topic model
    # wants a different matrix. LDA is only defined over integer counts, NMF
    # conventionally wants tf-idf, MEM takes either plus one-hot, and each wants
    # its own vocabulary size. worse, the shared matrix's filename carried only
    # the weighting -- so two models asking for different vocabularies targeted
    # the same file and rebuilt over each other, and whichever ran last won.
    #
    # they go in a folder named after the results' own stem, beside them,
    # rather than into a temp directory: somebody reading a topic model wants
    # to see which vocabulary the themes came out of.
    # the matrix folder is derived from the output's *stem*, not from its
    # folder. a fixed-name sibling (`parent / "matrix"`) is the same path for
    # every step that writes into one folder, and that produced the two worst
    # bugs in this feature: three topic models building over each other's
    # matrix, and an apply leaving a frozen vocabulary where a later fit picked
    # it up and modeled the wrong corpus. Deriving from the stem makes both
    # impossible rather than merely wired-around -- and it is the idiom the
    # rest of the codebase already uses for companion files.
    matrix_dir = out_features_csv.with_name(f"{stem}_matrix")
    matrix_dir.mkdir(parents=True, exist_ok=True)

    from .analyze_ngram_frequencies import analyze_ngram_frequencies
    from .build_doc_term_matrix import build_doc_term_matrix

    text_settings = dict(
        lemmatize=lemmatize, pos_tagged=pos_tagged, engine=engine,
        tokenizer=tokenizer, stanza_lang=stanza_lang,
        keep_punctuation=keep_punctuation, device=device)

    announce(on_progress, "counting the vocabulary")
    freq_list_csv = analyze_ngram_frequencies(
        analysis_csv=analysis_ready,
        out_features_csv=matrix_dir / "freq_list.csv",
        overwrite_existing=overwrite_existing, workers=workers,
        on_progress=on_progress, encoding=encoding,
        ngram_n=ngram_n, stoplist_paths=stoplist_paths,
        min_freq=min_freq, min_obs_pct=min_obs_pct,
        min_token_count=min_token_count, min_npmi=min_npmi,
        **text_settings)

    announce(on_progress, "building the matrix")
    dtm_csv = Path(build_doc_term_matrix(
        freq_list_csv=freq_list_csv, analysis_csv=analysis_ready,
        out_features_csv=matrix_dir / "dtm.csv",
        overwrite_existing=overwrite_existing, workers=workers,
        on_progress=on_progress, encoding=encoding,
        weighting=weighting, rounding=matrix_rounding,
        vocab_min_freq=vocab_min_freq, vocab_min_obs_pct=vocab_min_obs_pct,
        vocab_rule=vocab_rule, vocab_top_n=vocab_top_n,
        vocab_rank_by=vocab_rank_by,
        **text_settings))

    vocab = _load_vocabulary(
        Path(freq_list_csv), encoding=encoding, pos_tagged=pos_tagged,
        vocab_rule=vocab_rule,
        vocab_min_freq=vocab_min_freq, vocab_min_obs_pct=vocab_min_obs_pct,
        vocab_top_n=vocab_top_n, vocab_rank_by=vocab_rank_by,
    )
    terms = sorted(vocab,
                   key=lambda g: (-vocab[g]["frequency"], words_of(g), tags_of(g)))
    columns = column_names(terms, pos_tagged)
    _topics.check_matrix_agrees(dtm_csv, columns, encoding=encoding)

    # 2) one streaming pass to get the moments, then we find the axes in memory
    warn_if_wide(len(terms))
    n, sums, cross = stream_moments(dtm_csv, encoding=encoding, skip_cols=2,
                                    on_progress=on_progress,
                                    message="measuring the matrix")
    if n < 3:
        raise ValueError(f"Only {n} document(s) in {dtm_csv}; MEM needs a corpus.")
    announce(on_progress, "extracting themes")
    # the expensive half once, whatever happens next: a sweep over theme
    # counts is then a varimax per count rather than a decomposition per count
    kept, mu, sigma, eigvals, eigvecs = _decompose(n, sums, cross)
    if n_components == 0 and k_selection in _topics.K_SELECTION_RULES:
        import numpy as np

        def fit_at(candidate):
            themes, _proj, _eig, _pct = _axes_at(
                kept, eigvals, eigvecs, candidate, rotation=rotation)
            # a theme's words are its *positive* pole, put back where the
            # whole vocabulary can see them: `kept` dropped the constant
            # columns, and the co-occurrence counts are indexed over every
            # term. Getting this wrong scores a theme against other words
            # entirely, with no error anywhere.
            weights = np.zeros((candidate, len(terms)))
            weights[:, kept] = np.maximum(themes.T, 0.0)
            return weights

        def batches():
            return _topics.stream_counts(dtm_csv, encoding=encoding, skip_cols=2)

        n_components, _winner = _topics.select_k(
            k_values=k_values, fit=fit_at, terms=terms, batches=batches,
            engine="mem", rule=k_selection, metric=coherence_metric,
            top_terms=top_terms, rounding=rounding,
            out_stem=out_features_csv.with_name(f"{stem}_k_selection"),
            encoding=encoding, on_progress=on_progress)
        retention = {"rule": k_selection, "metric": coherence_metric,
                     "n_components": int(n_components)}
        k = n_components
    else:
        k, retention = _retain_count(
            n, kept.size, eigvals, n_components=n_components,
            retain=k_selection, kaiser_cutoff=kaiser_cutoff,
            on_progress=on_progress)
    loadings, projection, theme_eig, pct = _axes_at(
        kept, eigvals, eigvecs, k, rotation=rotation)
    theme_names = [f"Theme_{i + 1}" for i in range(k)]

    kept_set = set(kept.tolist())
    dropped = [columns[j] for j in range(len(terms)) if j not in kept_set]
    if dropped:
        import warnings
        warnings.warn(
            f"{len(dropped)} constant term column(s) carried no signal and "
            f"were left out of the model: {', '.join(dropped[:8])}"
            f"{'…' if len(dropped) > 8 else ''}")

    # 3) the loadings and eigenvalue tables (small, so we do them in memory)
    kept_terms = [terms[j] for j in kept.tolist()]
    with atomic_write(loadings_path, newline="", encoding=encoding) as f:
        writer = csv.writer(f)
        head = ["term", "pos", *theme_names] if pos_tagged else ["term", *theme_names]
        writer.writerow(head)
        for i, gram in enumerate(kept_terms):
            row = [words_of(gram)]
            if pos_tagged:
                row.append(tags_of(gram))
            row.extend(round(float(v), rounding) for v in loadings[i])
            writer.writerow(row)
    # Two quantities, two tables. They were one table with a shared `rank`
    # column, which implied theme 3 was built from eigenvector 3 -- and it is
    # not. Varimax rotates the kept axes within the space they span, so each
    # rotated theme is a remix of all of them. Both lists are sorted
    # descending and that is the only thing they have in common, which is
    # exactly the kind of coincidence a shared column turns into a claim.

    # the spectrum: what the retention rule read, and the curve to judge
    # signal by. Every rank, not just the kept ones -- seeing where the curve
    # crosses the chance line is the whole reason to look at it.
    thresholds = list(retention.get("thresholds") or [])
    with atomic_write(eigen_path, newline="", encoding=encoding) as f:
        writer = csv.writer(f)
        writer.writerow(["rank", "eigenvalue", "chance_threshold", "kept"])
        for i, value in enumerate(eigvals):
            writer.writerow([
                i + 1, round(float(value), rounding),
                round(float(thresholds[i]), rounding) if i < len(thresholds) else "",
                "yes" if i < k else ""])

    # and what each theme actually accounts for. Not an eigenvalue: a rotated
    # theme is not an eigenvector, so it has none. This is its sum of squared
    # loadings, which is the same quantity *before* rotation and deliberately
    # redistributed by it -- which is why this column is much flatter than the
    # spectrum, and why reading it as a scree curve looks like broken math.
    with atomic_write(variance_path, newline="", encoding=encoding) as f:
        writer = csv.writer(f)
        writer.writerow(["theme", "variance", "pct_variance"])
        for name, eig, p_ in zip(theme_names, theme_eig, pct):
            writer.writerow([name, round(float(eig), rounding),
                             round(float(p_), rounding)])

    # 4) the model itself: everything apply needs and nothing it doesn't.
    #    `device` is a runtime choice, not part of the instrument, so we do
    #    NOT store it
    from datetime import date

    def _version() -> str:
        try:
            from importlib.metadata import version
            return version("taters")
        except Exception:
            return ""

    model = {
        "kind": "taters-mem-model",
        "format": MODEL_FORMAT,
        "created": date.today().isoformat(),
        "taters": _version(),
        "text": {"lemmatize": lemmatize, "pos_tagged": pos_tagged,
                 "engine": engine, "tokenizer": tokenizer,
                 "stanza_lang": stanza_lang,
                 "keep_punctuation": keep_punctuation},
        "matrix": {"weighting": weighting, "rounding": matrix_rounding,
                   "terms": terms, "columns": columns,
                   "idf": [vocab[g]["idf"] for g in terms]},
        "model": {"n_documents": n, "rotation": bool(rotation),
                  "retention": retention,
                  "themes": theme_names,
                  "kept": kept.tolist(),
                  "mu": mu.tolist(), "sigma": sigma.tolist(),
                  "projection": projection.tolist(),
                  "eigenvalues": theme_eig.tolist(),
                  "pct_variance": pct.tolist()},
    }
    with atomic_write(model_path, encoding="utf-8") as f:
        json.dump(model, f, indent=1)

    # 5) lastly, a second streaming pass: we score every document through the
    #    shared projection helper (the very same one apply uses)
    ticker = Ticker(on_progress, n)
    with dtm_csv.open("r", newline="", encoding=encoding) as f, \
            atomic_write(out_features_csv, newline="", encoding=encoding) as out:
        reader = csv.reader(f)
        next(reader)
        writer = csv.writer(out)
        writer.writerow(["text_id", "token_count", *theme_names])
        for row in reader:
            ticker.tick(message="scoring themes")
            writer.writerow(_theme_row(row[0], row[1], row[2:], kept, mu,
                                       sigma, projection, rounding))

    return out_features_csv

taters.text.topic_model_lda

Latent Dirichlet Allocation: the topic model most papers mean by "topic model".

What it does

LDA tells a story about how a corpus got written. Every document is a mixture of topics -- eighty percent about food, twenty about work -- and every topic is a distribution over words. Fitting runs that story backwards: given the words that actually appeared, what mixtures and what topics would best explain them?

You get back, for each document, the proportion of it that belongs to each topic. Those proportions sum to one, which makes them read naturally as "this interview was mostly about X" and makes them awkward as ordinary predictors -- see the note on that below.

How it differs from MEM

Taters already has a topic model, :mod:taters.text.topic_model_mem, and they answer different questions. MEM is PCA with a rotation: it finds the dimensions along which word use co-varies, and a document gets a score on each, positive or negative. LDA is generative and non-negative: it finds distributions over words, and a document gets a share of each. MEM's themes are contrasts; LDA's topics are ingredients. Neither is the better one, and a study that reports both is not doing the same thing twice.

A note about MALLET

If you have used LDA through DLATK, you have used MALLET -- a Java program, driven through a gensim wrapper that gensim deleted in version 4. Nothing here shells out to Java. This is variational Bayes (Hoffman, Blei & Bach 2010), which is the same family scikit-learn and gensim's own LdaModel use, and it is not the same algorithm as MALLET's collapsed Gibbs sampling. Topics will be comparable in character; the numbers will not match, and nothing here pretends they do.

Counts, and only counts

LDA's story is about how many times a word was said. Handed a tf-idf matrix it runs perfectly happily and returns numbers that mean nothing at all, so this refuses any weighting but count, by name, before it does any work. NMF is the one that wants tf-idf; see :mod:taters.text.topic_model_nmf.

topic_model_lda

topic_model_lda(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    workers=0,
    device="auto",
    on_progress=None,
    out_features_csv=None,
    out_model_json=None,
    out_loadings_csv=None,
    out_top_terms_csv=None,
    overwrite_existing=False,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    ngram_n=1,
    stoplist_paths=None,
    min_freq=5,
    min_obs_pct=0.1,
    min_token_count=10,
    min_npmi=None,
    lemmatize=False,
    pos_tagged=False,
    engine="nltk",
    tokenizer="potts",
    stanza_lang="en",
    keep_punctuation=False,
    weighting="count",
    matrix_rounding=4,
    vocab_min_freq=0,
    vocab_min_obs_pct=0,
    vocab_rule="top_n",
    vocab_top_n=2000,
    vocab_rank_by="obs_pct",
    n_topics=20,
    k_selection="coherence_exclusivity",
    k_values=_topics.DEFAULT_K_VALUES,
    coherence_metric="npmi",
    alpha=0.1,
    eta=0.01,
    passes=10,
    seed=42,
    top_terms=15,
    rounding=4
)

Fit LDA topics to a corpus; write per-document proportions and a model.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse.	None
txt_dir	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse.	None
analysis_csv	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse.	None
gathered_csv	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse.	None
workers	int	Worker processes for gathering, counting and scoring. 0 picks a sensible number for the machine and the job.	0
device	('auto', 'cuda', 'cpu')	Where Stanza runs, when engine="stanza". The fit itself is CPU arithmetic and does not use a GPU.	"auto"
out_features_csv	str or Path	Per-document topic proportions. Defaults to ./features/topic_model_lda/<gathered filename>. The model, the loadings and the top-terms table are written beside it unless given their own paths.	None
out_model_json	str or Path	Where the reusable model, the term-by-topic table the word clouds are drawn from, and the readable top-terms summary go.	None
out_loadings_csv	str or Path	Where the reusable model, the term-by-topic table the word clouds are drawn from, and the readable top-terms summary go.	None
out_top_terms_csv	str or Path	Where the reusable model, the term-by-topic table the word clouds are drawn from, and the readable top-terms summary go.	None
overwrite_existing	bool	If False and the output exists, skip the work and return the path.	False
encoding	str	Encoding for reading and writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	When gathering from a CSV, name(s) of the column(s) containing text.	("text",)
id_cols	Sequence[str] or None	Optional ID columns that identify each row when gathering from CSV.	None
mode	('concat', 'separate')	Whether multiple text columns are joined into one document or measured separately.	"concat"
group_by	Sequence[str] or None	Optional grouping keys used during CSV gathering -- one document per group instead of one per row.	None
pattern	str	Which files to read when gathering from a folder. Only with txt_dir.	every document type
ngram_n	int	Highest n-gram order to consider for the vocabulary.	1
stoplist_paths	Sequence[str or Path] or None	Word lists to drop before counting. Worth using: function words are the most frequent words in any corpus and a topic made of them tells you nothing.	None
min_freq	int	Drop terms rarer than this from the vocabulary.	5
min_obs_pct	float	Drop terms appearing in fewer than this percent of documents.	0.10
min_token_count	int	Skip documents shorter than this many tokens entirely.	10
min_npmi	float	Optional collocation threshold for orders above one.	None
lemmatize	bool	Lemmatize before counting, so "run" and "running" are one term.	False
pos_tagged	bool	Keep part-of-speech tags on terms, so "book/NOUN" and "book/VERB" are different terms.	False
engine	('nltk', 'stanza')	Which tokenizer and tagger to use.	"nltk"
tokenizer	('potts', 'stanza')	Which tokenizer rules to apply.	"potts"
stanza_lang	str	Language for the Stanza engine.	"en"
keep_punctuation	bool	Count punctuation as terms.	False
weighting	'count'	Counts only. LDA's generative story is about how many times a word was said, so it is only defined over integers. Given a tfidf or relfreq matrix it would run and return meaningless numbers, so anything else is refused. For tf-idf, use NMF.	"count"
matrix_rounding	int	Decimal places in the written matrix.	4
vocab_min_freq	float	How the vocabulary is cut from the frequency list. vocab_top_n defaults to 2000 here rather than MEM's 500: LDA has more topics to separate and starves on a small vocabulary.	0
vocab_min_obs_pct	float	How the vocabulary is cut from the frequency list. vocab_top_n defaults to 2000 here rather than MEM's 500: LDA has more topics to separate and starves on a small vocabulary.	0
vocab_rule	float	How the vocabulary is cut from the frequency list. vocab_top_n defaults to 2000 here rather than MEM's 500: LDA has more topics to separate and starves on a small vocabulary.	0
vocab_top_n	float	How the vocabulary is cut from the frequency list. vocab_top_n defaults to 2000 here rather than MEM's 500: LDA has more topics to separate and starves on a small vocabulary.	0
vocab_rank_by	('obs_pct', 'frequency')	What top_n ranks by when it cuts the vocabulary. obs_pct is the share of documents a term appears in; frequency is its raw count. Spread, not volume, is what a topic model needs: a word one document repeats five hundred times outranks everything on frequency and cannot distinguish a thing, because it only ever describes that one document. A word used once each across half the corpus is what topics are made of. (The standalone document-term matrix still ranks by frequency, because a feature table usually does want the commonest terms.)	"obs_pct"
n_topics	int	How many topics to fit. 0 chooses for you by k_selection, which costs one fit per candidate count and is the expensive path -- see the <stem>_k_selection files it leaves behind for the evidence.	20
k_selection	('coherence', 'coherence_exclusivity')	How 0 chooses. coherence maximizes how often a topic's top words turn up in the same documents. coherence_exclusivity takes the harmonic mean of that and exclusivity -- whether those are this topic's words rather than everybody's -- because coherence alone is maximized by a few topics made of common words. Needs coherence_metric="npmi".	"coherence"
k_values	str or sequence of int	The counts to try. "5,10,20" as well as a list, since this arrives from a pipeline file and from a command line. Counts the corpus is too small to support are skipped and named rather than ending the run.	DEFAULT_K_VALUES
coherence_metric	('npmi', 'umass')	Which coherence. NPMI is bounded, which is what lets it be balanced against exclusivity without rescaling.	"npmi"
alpha	float	Prior on the document-topic mixtures. Smaller makes each document commit to fewer topics.	0.1
eta	float	Prior on the topic-word distributions. Smaller makes each topic commit to fewer words.	0.01
passes	int	How many times to walk the corpus.	10
seed	int	Fixes the initialization, and with it the whole fit. Recorded in the model, so a run can be repeated exactly.	42
top_terms	int	How many words to list per topic in the readable summary.	15
rounding	int	Decimal places in the written proportions and loadings.	4

Returns:

Type	Description
Path	out_features_csv: text_id, token_count, then Topic_1..Topic_k, each row summing to 1.

Notes

Topic proportions sum to one for every document, so the last topic is one minus all the others and carries nothing the rest do not. Put every topic into a single regression and there is no unique answer -- many different sets of coefficients fit identically.

The field's answer to this is not a transform, it is the pipeline. Schwartz et al. (2013), who put 2,000 LDA topics in front of personality outcomes, ran a separate regression per feature with covariates for interpretation, and reduced with PCA before the ridge for prediction. Taters' correlations and group differences already work one measure at a time; for the ridge, turn on the analysis step's pca setting. PCA also makes the sum-to-one problem vanish on its own -- the redundant direction has zero variance, so it is dropped without anybody having to think about it.

Neither MEM's themes nor NMF's factors are constrained this way.

Source code in src\taters\text\topic_model_lda.py

@records_settings(
    binding=TEXT_INPUT, grain=TEXT_GRAIN,
    outputs=("out_features_csv", "out_model_json", "out_loadings_csv",
             "out_top_terms_csv"),
    # the topics are fitted to this corpus, so the honest way to measure them on
    # another is to apply the saved model rather than to fit again. refitting
    # gives a second study its own topics in its own order, and a ridge trained
    # on the first one cannot be scored with them at all.
    replay=(f"{__name__}:apply_lda_model", {"model_json": "out_model_json"}),
    bookkeeping=("token_count",))
def topic_model_lda(
    *,
    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    workers: int = 0,
    device: str = "auto",
    on_progress: Optional[Callable[..., None]] = None,

    # ----- Output -----
    out_features_csv: Optional[PathLike] = None,
    out_model_json: Optional[PathLike] = None,
    out_loadings_csv: Optional[PathLike] = None,
    out_top_terms_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ====== TXT FOLDER GATHER OPTIONS ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ----- the vocabulary this model builds for itself -----
    ngram_n: int = 1,
    stoplist_paths: Optional[Sequence[PathLike]] = None,
    min_freq: int = 5,
    min_obs_pct: float = 0.10,
    min_token_count: int = 10,
    min_npmi: Optional[float] = None,
    lemmatize: bool = False,
    pos_tagged: bool = False,
    engine: Literal["nltk", "stanza"] = "nltk",
    tokenizer: Literal["potts", "stanza"] = "potts",
    stanza_lang: str = "en",
    keep_punctuation: bool = False,
    weighting: Literal["count"] = "count",
    matrix_rounding: int = 4,
    vocab_min_freq: float = 0,
    vocab_min_obs_pct: float = 0,
    vocab_rule: Literal["top_n", "min_obs_pct", "min_freq"] = "top_n",
    vocab_top_n: int = 2000,
    vocab_rank_by: Literal["obs_pct", "frequency"] = "obs_pct",

    # ----- LDA options -----
    n_topics: int = 20,
    k_selection: Literal["coherence", "coherence_exclusivity"] = "coherence_exclusivity",
    k_values: Union[str, Sequence[int]] = _topics.DEFAULT_K_VALUES,
    coherence_metric: Literal["npmi", "umass"] = "npmi",
    alpha: float = 0.1,
    eta: float = 0.01,
    passes: int = 10,
    seed: int = 42,
    top_terms: int = 15,
    rounding: int = 4,
) -> Path:
    """
    Fit LDA topics to a corpus; write per-document proportions and a model.

    Parameters
    ----------
    csv_path, txt_dir, analysis_csv, gathered_csv
        The corpus, given exactly one of these ways: a spreadsheet, a folder of
        documents, an already-gathered analysis-ready table, or a gathered
        table to write and reuse.
    workers : int, default=0
        Worker processes for gathering, counting and scoring. 0 picks a
        sensible number for the machine and the job.
    device : {"auto", "cuda", "cpu"}, default="auto"
        Where Stanza runs, when ``engine="stanza"``. The fit itself is CPU
        arithmetic and does not use a GPU.
    out_features_csv : str or pathlib.Path, optional
        Per-document topic proportions. Defaults to
        ``./features/topic_model_lda/<gathered filename>``. The model, the
        loadings and the top-terms table are written beside it unless given
        their own paths.
    out_model_json, out_loadings_csv, out_top_terms_csv : str or Path, optional
        Where the reusable model, the term-by-topic table the word clouds are
        drawn from, and the readable top-terms summary go.
    overwrite_existing : bool, default=False
        If ``False`` and the output exists, skip the work and return the path.
    encoding : str, default="utf-8-sig"
        Encoding for reading and writing CSV files.
    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV, name(s) of the column(s) containing text.
    id_cols : Sequence[str] or None, optional
        Optional ID columns that identify each row when gathering from CSV.
    mode : {"concat", "separate"}, default="concat"
        Whether multiple text columns are joined into one document or measured
        separately.
    group_by : Sequence[str] or None, optional
        Optional grouping keys used during CSV gathering -- one document per
        group instead of one per row.
    pattern : str, default=every document type
        Which files to read when gathering from a folder. Only with ``txt_dir``.
    ngram_n : int, default=1
        Highest n-gram order to consider for the vocabulary.
    stoplist_paths : Sequence[str or pathlib.Path] or None, optional
        Word lists to drop before counting. Worth using: function words are
        the most frequent words in any corpus and a topic made of them tells
        you nothing.
    min_freq : int, default=5
        Drop terms rarer than this from the vocabulary.
    min_obs_pct : float, default=0.10
        Drop terms appearing in fewer than this *percent* of documents.
    min_token_count : int, default=10
        Skip documents shorter than this many tokens entirely.
    min_npmi : float, optional
        Optional collocation threshold for orders above one.
    lemmatize : bool, default=False
        Lemmatize before counting, so "run" and "running" are one term.
    pos_tagged : bool, default=False
        Keep part-of-speech tags on terms, so "book/NOUN" and "book/VERB" are
        different terms.
    engine : {"nltk", "stanza"}, default="nltk"
        Which tokenizer and tagger to use.
    tokenizer : {"potts", "stanza"}, default="potts"
        Which tokenizer rules to apply.
    stanza_lang : str, default="en"
        Language for the Stanza engine.
    keep_punctuation : bool, default=False
        Count punctuation as terms.
    weighting : {"count"}, default="count"
        **Counts only.** LDA's generative story is about how many times a word
        was said, so it is only defined over integers. Given a ``tfidf`` or
        ``relfreq`` matrix it would run and return meaningless numbers, so
        anything else is refused. For tf-idf, use NMF.
    matrix_rounding : int, default=4
        Decimal places in the written matrix.
    vocab_min_freq, vocab_min_obs_pct, vocab_rule, vocab_top_n
        How the vocabulary is cut from the frequency list. ``vocab_top_n``
        defaults to 2000 here rather than MEM's 500: LDA has more topics to
        separate and starves on a small vocabulary.
    vocab_rank_by : {"obs_pct", "frequency"}, default="obs_pct"
        What ``top_n`` ranks by when it cuts the vocabulary. ``obs_pct`` is the
        share of documents a term appears in; ``frequency`` is its raw count.

        Spread, not volume, is what a topic model needs: a word one document
        repeats five hundred times outranks everything on frequency and
        cannot distinguish a thing, because it only ever describes that one
        document. A word used once each across half the corpus is what topics
        are made of. (The standalone document-term matrix still ranks by
        frequency, because a *feature* table usually does want the commonest
        terms.)
    n_topics : int, default=20
        How many topics to fit. ``0`` chooses for you by ``k_selection``,
        which costs one fit per candidate count and is the expensive path --
        see the ``<stem>_k_selection`` files it leaves behind for the
        evidence.
    k_selection : {"coherence", "coherence_exclusivity"}, default="coherence_exclusivity"
        How ``0`` chooses. ``coherence`` maximizes how often a topic's top
        words turn up in the same documents. ``coherence_exclusivity`` takes
        the harmonic mean of that and exclusivity -- whether those are this
        topic's words rather than everybody's -- because coherence alone is
        maximized by a few topics made of common words. Needs
        ``coherence_metric="npmi"``.
    k_values : str or sequence of int
        The counts to try. ``"5,10,20"`` as well as a list, since this
        arrives from a pipeline file and from a command line. Counts the
        corpus is too small to support are skipped and named rather than
        ending the run.
    coherence_metric : {"npmi", "umass"}, default="npmi"
        Which coherence. NPMI is bounded, which is what lets it be balanced
        against exclusivity without rescaling.
    alpha : float, default=0.1
        Prior on the document-topic mixtures. Smaller makes each document
        commit to fewer topics.
    eta : float, default=0.01
        Prior on the topic-word distributions. Smaller makes each topic commit
        to fewer words.
    passes : int, default=10
        How many times to walk the corpus.
    seed : int, default=42
        Fixes the initialization, and with it the whole fit. Recorded in the
        model, so a run can be repeated exactly.
    top_terms : int, default=15
        How many words to list per topic in the readable summary.
    rounding : int, default=4
        Decimal places in the written proportions and loadings.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id``, ``token_count``, then
        ``Topic_1..Topic_k``, each row summing to 1.

    Notes
    -----
    Topic proportions sum to one for every document, so the last topic is one
    minus all the others and carries nothing the rest do not. Put every topic
    into a single regression and there is no unique answer -- many different
    sets of coefficients fit identically.

    The field's answer to this is not a transform, it is the pipeline. Schwartz
    et al. (2013), who put 2,000 LDA topics in front of personality outcomes,
    ran *a separate regression per feature* with covariates for interpretation,
    and reduced with PCA before the ridge for prediction. Taters' correlations
    and group differences already work one measure at a time; for the ridge,
    turn on the analysis step's ``pca`` setting. PCA also makes the sum-to-one
    problem vanish on its own -- the redundant direction has zero variance, so
    it is dropped without anybody having to think about it.

    Neither MEM's themes nor NMF's factors are constrained this way.
    """
    if weighting != "count":
        raise ValueError(
            f"LDA needs counts, not {weighting!r}. Its model is of how many "
            "times each word was said, so a tf-idf or relative-frequency "
            "matrix would fit without complaint and mean nothing. Use "
            "weighting='count' here, or topic_model_nmf for tf-idf.")

    import numpy as np

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        out_features_csv = (Path.cwd() / "features" / "topic_model_lda"
                            / analysis_ready.name)
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)
    stem = out_features_csv.stem
    model_path = Path(out_model_json) if out_model_json else \
        out_features_csv.with_name(f"{stem}_model.json")
    loadings_path = Path(out_loadings_csv) if out_loadings_csv else \
        out_features_csv.with_name(f"{stem}_loadings.csv")
    terms_path = Path(out_top_terms_csv) if out_top_terms_csv else \
        out_features_csv.with_name(f"{stem}_top_terms.csv")

    if not overwrite_existing and out_features_csv.is_file():
        print("LDA topic output file already exists; returning existing file.")
        return out_features_csv

    text_settings = dict(lemmatize=lemmatize, pos_tagged=pos_tagged,
                         engine=engine, tokenizer=tokenizer,
                         stanza_lang=stanza_lang,
                         keep_punctuation=keep_punctuation, device=device)
    vocab_settings = dict(vocab_min_freq=vocab_min_freq,
                          vocab_min_obs_pct=vocab_min_obs_pct,
                          vocab_rule=vocab_rule, vocab_top_n=vocab_top_n,
                          vocab_rank_by=vocab_rank_by)
    freq_list_csv, dtm_csv, terms, columns, vocab = _topics.build_matrix(
    # the matrix folder is derived from the output's *stem*, not from its
    # folder. a fixed-name sibling (`parent / "matrix"`) is the same path for
    # every step that writes into one folder, and that produced the two worst
    # bugs in this feature: three topic models building over each other's
    # matrix, and an apply leaving a frozen vocabulary where a later fit picked
    # it up and modeled the wrong corpus. Deriving from the stem makes both
    # impossible rather than merely wired-around -- and it is the idiom the
    # rest of the codebase already uses for companion files.
        analysis_ready=analysis_ready,
        out_dir=out_features_csv.with_name(f"{stem}_matrix"),
        weighting=weighting, matrix_rounding=matrix_rounding,
        text_settings=text_settings, vocab_settings=vocab_settings,
        ngram_settings=dict(ngram_n=ngram_n, stoplist_paths=stoplist_paths,
                            min_freq=min_freq, min_obs_pct=min_obs_pct,
                            min_token_count=min_token_count, min_npmi=min_npmi),
        overwrite_existing=overwrite_existing, workers=workers,
        on_progress=on_progress, encoding=encoding)

    announce(on_progress, "fitting topics")

    def batches():
        return _topics.stream_counts(dtm_csv, encoding=encoding, skip_cols=2)

    def fit_at(k):
        return _topics.fit_lda(batches, len(terms), k, alpha=alpha, eta=eta,
                               passes=passes, seed=seed)[0]

    trace: list = []
    if n_topics:
        lam, trace = _topics.fit_lda(
            batches, len(terms), n_topics, alpha=alpha, eta=eta, passes=passes,
            seed=seed, on_progress=on_progress)
    else:
        # the sweep hands back the winner's own fit rather than making us do
        # the same deterministic work a second time
        n_topics, lam = _topics.select_k(
            k_values=k_values, fit=fit_at, terms=terms, batches=batches,
            engine="lda", rule=k_selection, metric=coherence_metric,
            top_terms=top_terms, rounding=rounding,
            out_stem=out_features_csv.with_name(f"{stem}_k_selection"),
            encoding=encoding, on_progress=on_progress)
    topic_names = _topic_names(n_topics)

    n_documents = sum(len(block) for block in batches())

    # the loadings table, in the shape `figures.wordclouds.theme_wordclouds`
    # already reads -- term, an optional pos, then one column per topic. it is
    # the same file for every topic model on purpose, so the word clouds are
    # written once and not once per engine.
    with atomic_write(loadings_path, newline="", encoding=encoding) as f:
        writer = csv.writer(f)
        head = ["term", "pos", *topic_names] if pos_tagged else ["term", *topic_names]
        writer.writerow(head)
        weights = lam / lam.sum(axis=1)[:, None]
        for j, gram in enumerate(terms):
            row = [words_of(gram)]
            if pos_tagged:
                row.append(tags_of(gram))
            row.extend(round(float(weights[i, j]), rounding)
                       for i in range(n_topics))
            writer.writerow(row)

    # and the readable one: what each topic is actually about, in order
    with atomic_write(terms_path, newline="", encoding=encoding) as f:
        writer = csv.writer(f)
        writer.writerow(["topic", "rank", "term", "weight"])
        weights = lam / lam.sum(axis=1)[:, None]
        for i, name in enumerate(topic_names):
            order = np.argsort(weights[i])[::-1][:top_terms]
            for rank, j in enumerate(order, start=1):
                writer.writerow([name, rank, words_of(terms[j]),
                                 round(float(weights[i, j]), rounding)])

    from datetime import date

    model = {
        "kind": MODEL_KIND,
        "format": MODEL_FORMAT,
        "created": date.today().isoformat(),
        "taters": _model_version(),
        # `device` is deliberately absent: where Stanza ran is a runtime choice,
        # not part of the instrument, and recording it would make two otherwise
        # identical models compare as different.
        "text": {"lemmatize": lemmatize, "pos_tagged": pos_tagged,
                 "engine": engine, "tokenizer": tokenizer,
                 "stanza_lang": stanza_lang,
                 "keep_punctuation": keep_punctuation},
        "matrix": {"weighting": weighting, "rounding": matrix_rounding,
                   "terms": terms, "columns": columns,
                   "idf": [vocab[g]["idf"] for g in terms]},
        "model": {"n_documents": n_documents, "topics": topic_names,
                  "alpha": alpha, "eta": eta, "passes": passes, "seed": seed,
                  "perplexity_trace": [round(float(x), 4) for x in trace],
                  "lambda": lam.tolist()},
    }
    with atomic_write(model_path, encoding="utf-8") as f:
        json.dump(model, f, indent=1)

    _score_matrix(dtm_csv, out_features_csv, lam=lam, alpha=alpha,
                  topic_names=topic_names, rounding=rounding,
                  encoding=encoding, on_progress=on_progress,
                  n_documents=n_documents)
    return out_features_csv

apply_lda_model

apply_lda_model(
    *,
    model_json,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    workers=0,
    device="auto",
    on_progress=None,
    out_features_csv=None,
    overwrite_existing=False,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    rounding=4
)

Score a corpus with topics fitted somewhere else.

Takes no vocabulary, tokenizer or weighting settings at all: they come out of the model, because the model is the instrument. That is what makes two studies comparable -- measure the second corpus with the first one's topics, rather than fitting new topics and hoping Topic_3 means the same thing.

Parameters:

Name	Type	Description	Default
model_json	str or Path	A model written by :func:topic_model_lda, or a library folder holding exactly one.	required
csv_path	Optional[PathLike]	The corpus to score, exactly one way.	None
txt_dir	Optional[PathLike]	The corpus to score, exactly one way.	None
analysis_csv	Optional[PathLike]	The corpus to score, exactly one way.	None
gathered_csv	Optional[PathLike]	The corpus to score, exactly one way.	None
workers	int	Worker processes. 0 picks a sensible number.	0
device	('auto', 'cuda', 'cpu')	Where Stanza runs, if the model was built with it.	"auto"
out_features_csv	str or Path	Defaults to ./features/topic_model_lda/<gathered filename>.	None
overwrite_existing	bool	If False and the output exists, return it unchanged.	False
encoding	str	Encoding for reading and writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	Gathering options, as in :func:topic_model_lda.	('text',)
id_cols	Sequence[str]	Gathering options, as in :func:topic_model_lda.	('text',)
mode	Sequence[str]	Gathering options, as in :func:topic_model_lda.	('text',)
group_by	Sequence[str]	Gathering options, as in :func:topic_model_lda.	('text',)
pattern	Sequence[str]	Gathering options, as in :func:topic_model_lda.	('text',)
rounding	int	Decimal places in the written proportions.	4

Returns:

Type	Description
Path	out_features_csv: text_id, token_count, then one column per topic of the model.

Source code in src\taters\text\topic_model_lda.py

@records_settings(
    # the model file is the whole instrument -- vocabulary, tokenizer settings
    # and topics -- so it is compared by content. two models with the same name
    # and different topics have to come out as different measurements.
    binding=TEXT_INPUT, grain=TEXT_GRAIN, assets={"model_json": None},
    outputs=("out_features_csv",), bookkeeping=("token_count",))
def apply_lda_model(
    *,
    model_json: PathLike,

    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    workers: int = 0,
    device: str = "auto",
    on_progress: Optional[Callable[..., None]] = None,

    # ----- Output -----
    out_features_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ====== TXT FOLDER GATHER OPTIONS ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    rounding: int = 4,
) -> Path:
    """
    Score a corpus with topics fitted somewhere else.

    Takes no vocabulary, tokenizer or weighting settings at all: they come out
    of the model, because the model *is* the instrument. That is what makes two
    studies comparable -- measure the second corpus with the first one's topics,
    rather than fitting new topics and hoping Topic_3 means the same thing.

    Parameters
    ----------
    model_json : str or pathlib.Path
        A model written by :func:`topic_model_lda`, or a library folder holding
        exactly one.
    csv_path, txt_dir, analysis_csv, gathered_csv
        The corpus to score, exactly one way.
    workers : int, default=0
        Worker processes. 0 picks a sensible number.
    device : {"auto", "cuda", "cpu"}, default="auto"
        Where Stanza runs, if the model was built with it.
    out_features_csv : str or pathlib.Path, optional
        Defaults to ``./features/topic_model_lda/<gathered filename>``.
    overwrite_existing : bool, default=False
        If ``False`` and the output exists, return it unchanged.
    encoding : str, default="utf-8-sig"
        Encoding for reading and writing CSV files.
    text_cols, id_cols, mode, group_by, pattern
        Gathering options, as in :func:`topic_model_lda`.
    rounding : int, default=4
        Decimal places in the written proportions.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id``, ``token_count``, then one column per
        topic of the model.
    """
    import numpy as np

    model = _load_model(one_model_path(model_json))
    text_cfg = model["text"]
    matrix_cfg = model["matrix"]
    fit = model["model"]

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        # `_applied`, so that fitting and applying with the defaults do not
        # write to one file -- and so their matrix folders, derived from this
        # stem, can never be the same one.
        out_features_csv = (Path.cwd() / "features" / "topic_model_lda"
                            / f"{analysis_ready.stem}_applied{analysis_ready.suffix}")
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)
    if not overwrite_existing and out_features_csv.is_file():
        print("LDA topic output file already exists; returning existing file.")
        return out_features_csv

    # rebuild the matrix the model's way, not the caller's. the vocabulary is
    # pinned to the model's own term list, so a word the training corpus never
    # saw simply does not count -- which is the honest thing: the model has no
    # topic for it.
    from .build_doc_term_matrix import build_doc_term_matrix

    # the apply gets its own matrix folder, never the fit's: the frozen
    # frequency list written below is deliberately *not* provenance-
    # recorded -- it is the model's vocabulary, not a measurement of this
    # corpus. Sharing one folder meant a later fit found that file, saw no
    # record to disagree with, reused it, and modeled the applied model's
    # vocabulary instead of its own corpus's. Fit on A, apply to B, fit on B,
    # and the second model came out holding A's words, with no error anywhere.
    # the matrix folder is derived from the output's *stem*, not from its
    # folder. a fixed-name sibling (`parent / "matrix"`) is the same path for
    # every step that writes into one folder, and that produced the two worst
    # bugs in this feature: three topic models building over each other's
    # matrix, and an apply leaving a frozen vocabulary where a later fit picked
    # it up and modeled the wrong corpus. Deriving from the stem makes both
    # impossible rather than merely wired-around -- and it is the idiom the
    # rest of the codebase already uses for companion files.
    matrix_dir = out_features_csv.with_name(
        f"{out_features_csv.stem}_matrix")
    matrix_dir.mkdir(parents=True, exist_ok=True)
    dtm_csv = Path(build_doc_term_matrix(
        freq_list_csv=_frozen_freq_list(model, matrix_dir, encoding=encoding),
        analysis_csv=analysis_ready,
        out_features_csv=matrix_dir / "dtm.csv",
        overwrite_existing=True, workers=workers, on_progress=on_progress,
        encoding=encoding, weighting=matrix_cfg["weighting"],
        rounding=int(matrix_cfg["rounding"]),
        vocab_min_freq=0, vocab_min_obs_pct=0, vocab_rule="top_n",
        vocab_top_n=len(matrix_cfg["terms"]), vocab_rank_by="frequency",
        lemmatize=text_cfg["lemmatize"], pos_tagged=text_cfg["pos_tagged"],
        engine=text_cfg["engine"], tokenizer=text_cfg["tokenizer"],
        stanza_lang=text_cfg["stanza_lang"],
        keep_punctuation=bool(text_cfg.get("keep_punctuation", True)),
        device=device))

    lam = np.asarray(fit["lambda"], dtype=np.float64)
    n_documents = sum(1 for _ in Path(dtm_csv).open(encoding=encoding)) - 1
    _score_matrix(dtm_csv, out_features_csv, lam=lam,
                  alpha=float(fit.get("alpha", 0.1)),
                  topic_names=list(fit["topics"]), rounding=rounding,
                  encoding=encoding, on_progress=on_progress,
                  n_documents=max(n_documents, 0))
    return out_features_csv

taters.text.topic_model_nmf

Non-negative matrix factorization: topics without the probability story.

What it does

NMF asks something simpler than LDA. Split the document-term matrix into two non-negative pieces -- documents by topics, topics by terms -- whose product is as close to the original as it can get. No generative story, no priors, nothing that has to sum to one. A document's topic weights are just weights: one can be large without forcing another to be small.

That simplicity is why it is worth having beside LDA rather than instead of it. On short texts -- tweets, open-ended survey answers, single utterances -- LDA often struggles because there is not enough of each document to infer a mixture from, and NMF's topics come out sharper and easier to name. On long documents the two tend to agree.

Which weighting, and why it differs from LDA

NMF defaults to tf-idf, and that is not an accident of taste: without it, the factorization spends its first topic on whatever words are simply common, because those are the cells with the most mass to explain. Down-weighting what is everywhere is how NMF gets topics rather than a frequency ranking.

LDA is the opposite -- it is only defined over integer counts, and refuses anything else. The two engines genuinely want different matrices, which is why each builds its own instead of sharing one.

What comes out

One column per topic, Factor_1..Factor_k. They are weights, not proportions: bigger means more of that topic, zero means none, and they do not add up to anything in particular. That makes them easier to use as ordinary predictors than LDA's proportions, which are compositional.

topic_model_nmf

topic_model_nmf(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    workers=0,
    device="auto",
    on_progress=None,
    out_features_csv=None,
    out_model_json=None,
    out_loadings_csv=None,
    out_top_terms_csv=None,
    overwrite_existing=False,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    ngram_n=1,
    stoplist_paths=None,
    min_freq=5,
    min_obs_pct=0.1,
    min_token_count=10,
    min_npmi=None,
    lemmatize=False,
    pos_tagged=False,
    engine="nltk",
    tokenizer="potts",
    stanza_lang="en",
    keep_punctuation=False,
    weighting="tfidf",
    matrix_rounding=4,
    vocab_min_freq=0,
    vocab_min_obs_pct=0,
    vocab_rule="top_n",
    vocab_top_n=2000,
    vocab_rank_by="obs_pct",
    n_topics=20,
    k_selection="coherence_exclusivity",
    k_values=_topics.DEFAULT_K_VALUES,
    coherence_metric="npmi",
    beta_loss="frobenius",
    iters=200,
    tol=0.0001,
    top_terms=15,
    rounding=4
)

Fit NMF factors to a corpus; write per-document weights and a model.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse.	None
txt_dir	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse.	None
analysis_csv	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse.	None
gathered_csv	Optional[PathLike]	The corpus, given exactly one of these ways: a spreadsheet, a folder of documents, an already-gathered analysis-ready table, or a gathered table to write and reuse.	None
workers	int	Worker processes for gathering, counting and scoring. 0 picks a sensible number for the machine and the job.	0
device	('auto', 'cuda', 'cpu')	Where Stanza runs, when engine="stanza". The fit itself is CPU arithmetic and does not use a GPU.	"auto"
out_features_csv	str or Path	Per-document topic proportions. Defaults to ./features/topic_model_nmf/<gathered filename>. The model, the loadings and the top-terms table are written beside it unless given their own paths.	None
out_model_json	str or Path	Where the reusable model, the term-by-topic table the word clouds are drawn from, and the readable top-terms summary go.	None
out_loadings_csv	str or Path	Where the reusable model, the term-by-topic table the word clouds are drawn from, and the readable top-terms summary go.	None
out_top_terms_csv	str or Path	Where the reusable model, the term-by-topic table the word clouds are drawn from, and the readable top-terms summary go.	None
overwrite_existing	bool	If False and the output exists, skip the work and return the path.	False
encoding	str	Encoding for reading and writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	When gathering from a CSV, name(s) of the column(s) containing text.	("text",)
id_cols	Sequence[str] or None	Optional ID columns that identify each row when gathering from CSV.	None
mode	('concat', 'separate')	Whether multiple text columns are joined into one document or measured separately.	"concat"
group_by	Sequence[str] or None	Optional grouping keys used during CSV gathering -- one document per group instead of one per row.	None
pattern	str	Which files to read when gathering from a folder. Only with txt_dir.	every document type
ngram_n	int	Highest n-gram order to consider for the vocabulary.	1
stoplist_paths	Sequence[str or Path] or None	Word lists to drop before counting. Worth using: function words are the most frequent words in any corpus and a topic made of them tells you nothing.	None
min_freq	int	Drop terms rarer than this from the vocabulary.	5
min_obs_pct	float	Drop terms appearing in fewer than this percent of documents.	0.10
min_token_count	int	Skip documents shorter than this many tokens entirely.	10
min_npmi	float	Optional collocation threshold for orders above one.	None
lemmatize	bool	Lemmatize before counting, so "run" and "running" are one term.	False
pos_tagged	bool	Keep part-of-speech tags on terms, so "book/NOUN" and "book/VERB" are different terms.	False
engine	('nltk', 'stanza')	Which tokenizer and tagger to use.	"nltk"
tokenizer	('potts', 'stanza')	Which tokenizer rules to apply.	"potts"
stanza_lang	str	Language for the Stanza engine.	"en"
keep_punctuation	bool	Count punctuation as terms.	False
weighting	('tfidf', 'count')	tf-idf by default, and that is a real recommendation rather than a shrug: without it the factorization spends its first factor on whatever words are merely common, because those cells hold the most mass. Raw counts are allowed for anyone who wants NMF and LDA over the same matrix. binary and relfreq are refused -- the first throws away how much a word was used, the second rescales every document to the same total, which is the thing the factorization is trying to see.	"tfidf"
matrix_rounding	int	Decimal places in the written matrix.	4
vocab_min_freq	float	How the vocabulary is cut from the frequency list. vocab_top_n defaults to 2000 here rather than MEM's 500: more factors need more vocabulary to separate them.	0
vocab_min_obs_pct	float	How the vocabulary is cut from the frequency list. vocab_top_n defaults to 2000 here rather than MEM's 500: more factors need more vocabulary to separate them.	0
vocab_rule	float	How the vocabulary is cut from the frequency list. vocab_top_n defaults to 2000 here rather than MEM's 500: more factors need more vocabulary to separate them.	0
vocab_top_n	float	How the vocabulary is cut from the frequency list. vocab_top_n defaults to 2000 here rather than MEM's 500: more factors need more vocabulary to separate them.	0
vocab_rank_by	('obs_pct', 'frequency')	What top_n ranks by when it cuts the vocabulary. obs_pct is the share of documents a term appears in; frequency is its raw count. Spread, not volume, is what a topic model needs: a word one document repeats five hundred times outranks everything on frequency and cannot distinguish a thing, because it only ever describes that one document. A word used once each across half the corpus is what topics are made of. (The standalone document-term matrix still ranks by frequency, because a feature table usually does want the commonest terms.)	"obs_pct"
n_topics	int	How many topics to fit. 0 chooses for you by k_selection, which costs one fit per candidate count and is the expensive path -- see the <stem>_k_selection files it leaves behind for the evidence.	20
k_selection	('coherence', 'coherence_exclusivity')	How 0 chooses. coherence maximizes how often a topic's top words turn up in the same documents. coherence_exclusivity takes the harmonic mean of that and exclusivity -- whether those are this topic's words rather than everybody's -- because coherence alone is maximized by a few topics made of common words. Needs coherence_metric="npmi".	"coherence"
k_values	str or sequence of int	The counts to try. "5,10,20" as well as a list, since this arrives from a pipeline file and from a command line. Counts the corpus is too small to support are skipped and named rather than ending the run.	DEFAULT_K_VALUES
coherence_metric	('npmi', 'umass')	Which coherence. NPMI is bounded, which is what lets it be balanced against exclusivity without rescaling.	"npmi"
beta_loss	('frobenius', 'kullback-leibler')	What "close to the original" means. Frobenius is least squares: faster, steadier, and the usual choice. Kullback-Leibler matches the way counts actually vary and tends to give sparser, more readable factors on short texts -- worth trying if the Frobenius factors all look alike.	"frobenius"
iters	int	Most passes to make. It stops early when the error settles.	200
tol	float	How small a relative improvement counts as settled.	1e-4
top_terms	int	How many words to list per factor in the readable summary.	15
rounding	int	Decimal places in the written weights and loadings.	4

Returns:

Type	Description
Path	out_features_csv: text_id, token_count, then Factor_1..Factor_k.

Notes

There is no seed. NMF starts from NNDSVD, which is derived from the matrix's own singular vectors and is therefore deterministic -- two runs on one corpus give one answer, with nothing random to record.

The weights are not proportions. They do not sum to anything in particular, which makes them more straightforward as regression predictors than LDA's topic shares -- those are compositional, and using all of them at once is a known trap. Factor scale is arbitrary, though: doubling a factor in H and halving it in W is the same model, so compare a factor against itself across documents rather than against another factor.

Source code in src\taters\text\topic_model_nmf.py

@records_settings(
    binding=TEXT_INPUT, grain=TEXT_GRAIN,
    outputs=("out_features_csv", "out_model_json", "out_loadings_csv",
             "out_top_terms_csv"),
    # the topics are fitted to this corpus, so the honest way to measure them on
    # another is to apply the saved model rather than to fit again. refitting
    # gives a second study its own topics in its own order, and a ridge trained
    # on the first one cannot be scored with them at all.
    replay=(f"{__name__}:apply_nmf_model", {"model_json": "out_model_json"}),
    bookkeeping=("token_count",))
def topic_model_nmf(
    *,
    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    workers: int = 0,
    device: str = "auto",
    on_progress: Optional[Callable[..., None]] = None,

    # ----- Output -----
    out_features_csv: Optional[PathLike] = None,
    out_model_json: Optional[PathLike] = None,
    out_loadings_csv: Optional[PathLike] = None,
    out_top_terms_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ====== TXT FOLDER GATHER OPTIONS ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ----- the vocabulary this model builds for itself -----
    ngram_n: int = 1,
    stoplist_paths: Optional[Sequence[PathLike]] = None,
    min_freq: int = 5,
    min_obs_pct: float = 0.10,
    min_token_count: int = 10,
    min_npmi: Optional[float] = None,
    lemmatize: bool = False,
    pos_tagged: bool = False,
    engine: Literal["nltk", "stanza"] = "nltk",
    tokenizer: Literal["potts", "stanza"] = "potts",
    stanza_lang: str = "en",
    keep_punctuation: bool = False,
    weighting: Literal["tfidf", "count"] = "tfidf",
    matrix_rounding: int = 4,
    vocab_min_freq: float = 0,
    vocab_min_obs_pct: float = 0,
    vocab_rule: Literal["top_n", "min_obs_pct", "min_freq"] = "top_n",
    vocab_top_n: int = 2000,
    vocab_rank_by: Literal["obs_pct", "frequency"] = "obs_pct",

    # ----- NMF options -----
    n_topics: int = 20,
    k_selection: Literal["coherence", "coherence_exclusivity"] = "coherence_exclusivity",
    k_values: Union[str, Sequence[int]] = _topics.DEFAULT_K_VALUES,
    coherence_metric: Literal["npmi", "umass"] = "npmi",
    beta_loss: Literal["frobenius", "kullback-leibler"] = "frobenius",
    iters: int = 200,
    tol: float = 1e-4,
    top_terms: int = 15,
    rounding: int = 4,
) -> Path:
    """
    Fit NMF factors to a corpus; write per-document weights and a model.

    Parameters
    ----------
    csv_path, txt_dir, analysis_csv, gathered_csv
        The corpus, given exactly one of these ways: a spreadsheet, a folder of
        documents, an already-gathered analysis-ready table, or a gathered
        table to write and reuse.
    workers : int, default=0
        Worker processes for gathering, counting and scoring. 0 picks a
        sensible number for the machine and the job.
    device : {"auto", "cuda", "cpu"}, default="auto"
        Where Stanza runs, when ``engine="stanza"``. The fit itself is CPU
        arithmetic and does not use a GPU.
    out_features_csv : str or pathlib.Path, optional
        Per-document topic proportions. Defaults to
        ``./features/topic_model_nmf/<gathered filename>``. The model, the
        loadings and the top-terms table are written beside it unless given
        their own paths.
    out_model_json, out_loadings_csv, out_top_terms_csv : str or Path, optional
        Where the reusable model, the term-by-topic table the word clouds are
        drawn from, and the readable top-terms summary go.
    overwrite_existing : bool, default=False
        If ``False`` and the output exists, skip the work and return the path.
    encoding : str, default="utf-8-sig"
        Encoding for reading and writing CSV files.
    text_cols : Sequence[str], default=("text",)
        When gathering from a CSV, name(s) of the column(s) containing text.
    id_cols : Sequence[str] or None, optional
        Optional ID columns that identify each row when gathering from CSV.
    mode : {"concat", "separate"}, default="concat"
        Whether multiple text columns are joined into one document or measured
        separately.
    group_by : Sequence[str] or None, optional
        Optional grouping keys used during CSV gathering -- one document per
        group instead of one per row.
    pattern : str, default=every document type
        Which files to read when gathering from a folder. Only with ``txt_dir``.
    ngram_n : int, default=1
        Highest n-gram order to consider for the vocabulary.
    stoplist_paths : Sequence[str or pathlib.Path] or None, optional
        Word lists to drop before counting. Worth using: function words are
        the most frequent words in any corpus and a topic made of them tells
        you nothing.
    min_freq : int, default=5
        Drop terms rarer than this from the vocabulary.
    min_obs_pct : float, default=0.10
        Drop terms appearing in fewer than this *percent* of documents.
    min_token_count : int, default=10
        Skip documents shorter than this many tokens entirely.
    min_npmi : float, optional
        Optional collocation threshold for orders above one.
    lemmatize : bool, default=False
        Lemmatize before counting, so "run" and "running" are one term.
    pos_tagged : bool, default=False
        Keep part-of-speech tags on terms, so "book/NOUN" and "book/VERB" are
        different terms.
    engine : {"nltk", "stanza"}, default="nltk"
        Which tokenizer and tagger to use.
    tokenizer : {"potts", "stanza"}, default="potts"
        Which tokenizer rules to apply.
    stanza_lang : str, default="en"
        Language for the Stanza engine.
    keep_punctuation : bool, default=False
        Count punctuation as terms.
    weighting : {"tfidf", "count"}, default="tfidf"
        tf-idf by default, and that is a real recommendation rather than a
        shrug: without it the factorization spends its first factor on whatever
        words are merely common, because those cells hold the most mass. Raw
        counts are allowed for anyone who wants NMF and LDA over the same
        matrix. ``binary`` and ``relfreq`` are refused -- the first throws away
        how much a word was used, the second rescales every document to the
        same total, which is the thing the factorization is trying to see.
    matrix_rounding : int, default=4
        Decimal places in the written matrix.
    vocab_min_freq, vocab_min_obs_pct, vocab_rule, vocab_top_n
        How the vocabulary is cut from the frequency list. ``vocab_top_n``
        defaults to 2000 here rather than MEM's 500: more factors need more
        vocabulary to separate them.
    vocab_rank_by : {"obs_pct", "frequency"}, default="obs_pct"
        What ``top_n`` ranks by when it cuts the vocabulary. ``obs_pct`` is the
        share of documents a term appears in; ``frequency`` is its raw count.

        Spread, not volume, is what a topic model needs: a word one document
        repeats five hundred times outranks everything on frequency and
        cannot distinguish a thing, because it only ever describes that one
        document. A word used once each across half the corpus is what topics
        are made of. (The standalone document-term matrix still ranks by
        frequency, because a *feature* table usually does want the commonest
        terms.)
    n_topics : int, default=20
        How many topics to fit. ``0`` chooses for you by ``k_selection``,
        which costs one fit per candidate count and is the expensive path --
        see the ``<stem>_k_selection`` files it leaves behind for the
        evidence.
    k_selection : {"coherence", "coherence_exclusivity"}, default="coherence_exclusivity"
        How ``0`` chooses. ``coherence`` maximizes how often a topic's top
        words turn up in the same documents. ``coherence_exclusivity`` takes
        the harmonic mean of that and exclusivity -- whether those are this
        topic's words rather than everybody's -- because coherence alone is
        maximized by a few topics made of common words. Needs
        ``coherence_metric="npmi"``.
    k_values : str or sequence of int
        The counts to try. ``"5,10,20"`` as well as a list, since this
        arrives from a pipeline file and from a command line. Counts the
        corpus is too small to support are skipped and named rather than
        ending the run.
    coherence_metric : {"npmi", "umass"}, default="npmi"
        Which coherence. NPMI is bounded, which is what lets it be balanced
        against exclusivity without rescaling.
    beta_loss : {"frobenius", "kullback-leibler"}, default="frobenius"
        What "close to the original" means. Frobenius is least squares: faster,
        steadier, and the usual choice. Kullback-Leibler matches the way counts
        actually vary and tends to give sparser, more readable factors on short
        texts -- worth trying if the Frobenius factors all look alike.
    iters : int, default=200
        Most passes to make. It stops early when the error settles.
    tol : float, default=1e-4
        How small a relative improvement counts as settled.
    top_terms : int, default=15
        How many words to list per factor in the readable summary.
    rounding : int, default=4
        Decimal places in the written weights and loadings.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id``, ``token_count``, then
        ``Factor_1..Factor_k``.

    Notes
    -----
    There is no ``seed``. NMF starts from NNDSVD, which is derived from the
    matrix's own singular vectors and is therefore deterministic -- two runs on
    one corpus give one answer, with nothing random to record.

    The weights are not proportions. They do not sum to anything in particular,
    which makes them more straightforward as regression predictors than LDA's
    topic shares -- those are compositional, and using all of them at once is a
    known trap. Factor *scale* is arbitrary, though: doubling a factor in H and
    halving it in W is the same model, so compare a factor against itself across
    documents rather than against another factor.
    """
    if weighting not in ("tfidf", "count"):
        raise ValueError(
            f"NMF needs tf-idf or raw counts, not {weighting!r}. A binary or "
            "relative-frequency matrix factors without complaint and gives "
            "topics of something nobody asked about -- binary throws away how "
            "much a word was used, and relative frequency rescales every "
            "document to the same total, which is the one thing the "
            "factorization is trying to see.")

    import numpy as np

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        out_features_csv = (Path.cwd() / "features" / "topic_model_nmf"
                            / analysis_ready.name)
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)
    stem = out_features_csv.stem
    model_path = Path(out_model_json) if out_model_json else \
        out_features_csv.with_name(f"{stem}_model.json")
    loadings_path = Path(out_loadings_csv) if out_loadings_csv else \
        out_features_csv.with_name(f"{stem}_loadings.csv")
    terms_path = Path(out_top_terms_csv) if out_top_terms_csv else \
        out_features_csv.with_name(f"{stem}_top_terms.csv")

    if not overwrite_existing and out_features_csv.is_file():
        print("NMF factor output file already exists; returning existing file.")
        return out_features_csv

    text_settings = dict(lemmatize=lemmatize, pos_tagged=pos_tagged,
                         engine=engine, tokenizer=tokenizer,
                         stanza_lang=stanza_lang,
                         keep_punctuation=keep_punctuation, device=device)
    vocab_settings = dict(vocab_min_freq=vocab_min_freq,
                          vocab_min_obs_pct=vocab_min_obs_pct,
                          vocab_rule=vocab_rule, vocab_top_n=vocab_top_n,
                          vocab_rank_by=vocab_rank_by)
    freq_list_csv, dtm_csv, terms, columns, vocab = _topics.build_matrix(
    # the matrix folder is derived from the output's *stem*, not from its
    # folder. a fixed-name sibling (`parent / "matrix"`) is the same path for
    # every step that writes into one folder, and that produced the two worst
    # bugs in this feature: three topic models building over each other's
    # matrix, and an apply leaving a frozen vocabulary where a later fit picked
    # it up and modeled the wrong corpus. Deriving from the stem makes both
    # impossible rather than merely wired-around -- and it is the idiom the
    # rest of the codebase already uses for companion files.
        analysis_ready=analysis_ready,
        out_dir=out_features_csv.with_name(f"{stem}_matrix"),
        weighting=weighting, matrix_rounding=matrix_rounding,
        text_settings=text_settings, vocab_settings=vocab_settings,
        ngram_settings=dict(ngram_n=ngram_n, stoplist_paths=stoplist_paths,
                            min_freq=min_freq, min_obs_pct=min_obs_pct,
                            min_token_count=min_token_count, min_npmi=min_npmi),
        overwrite_existing=overwrite_existing, workers=workers,
        on_progress=on_progress, encoding=encoding)

    def batches():
        return _topics.stream_counts(dtm_csv, encoding=encoding, skip_cols=2)

    # NMF needs the whole matrix at once -- the multiplicative updates touch
    # every cell on every pass -- so unlike LDA it cannot stream. say how much
    # that will cost before spending minutes reading it in, not after.
    n_documents = sum(len(block) for block in batches())
    wanted = _topics.nmf_memory_gb(n_documents, len(terms),
                                   n_topics or max(_topics.parse_counts(k_values)))
    if wanted >= 1.0:
        import warnings
        warnings.warn(
            f"NMF holds the whole matrix in memory: about {wanted:.1f} GB for "
            f"{n_documents:,} documents by {len(terms):,} terms. Reduce "
            "`vocab_top_n` if that is more than this machine has.")

    announce(on_progress, "fitting factors")
    # read once, whether we fit once or twenty times
    matrix = _topics.read_matrix(dtm_csv, encoding=encoding, skip_cols=2)

    def fit_at(k):
        return _topics.fit_nmf(matrix, k, beta_loss=beta_loss, iters=iters,
                               tol=tol)[1]

    trace: list = []
    if n_topics:
        _w, h, trace = _topics.fit_nmf(matrix, n_topics, beta_loss=beta_loss,
                                       iters=iters, tol=tol,
                                       on_progress=on_progress)
    else:
        n_topics, h = _topics.select_k(
            k_values=k_values, fit=fit_at, terms=terms, batches=batches,
            engine="nmf", rule=k_selection, metric=coherence_metric,
            top_terms=top_terms, rounding=rounding,
            out_stem=out_features_csv.with_name(f"{stem}_k_selection"),
            encoding=encoding, on_progress=on_progress)
    topic_names = _topic_names(n_topics)

    # the loadings table, in the shape `figures.wordclouds.theme_wordclouds`
    # already reads -- term, an optional pos, then one column per topic. it is
    # the same file for every topic model on purpose, so the word clouds are
    # written once and not once per engine.
    with atomic_write(loadings_path, newline="", encoding=encoding) as f:
        writer = csv.writer(f)
        head = ["term", "pos", *topic_names] if pos_tagged else ["term", *topic_names]
        writer.writerow(head)
        # scaled to sum to one per factor purely so the numbers are readable
        # and comparable between factors. NMF's H carries no such constraint,
        # and the scale of a factor is arbitrary anyway -- W absorbs it.
        weights = h / (h.sum(axis=1)[:, None] + 1e-100)
        for j, gram in enumerate(terms):
            row = [words_of(gram)]
            if pos_tagged:
                row.append(tags_of(gram))
            row.extend(round(float(weights[i, j]), rounding)
                       for i in range(n_topics))
            writer.writerow(row)

    # and the readable one: what each topic is actually about, in order
    with atomic_write(terms_path, newline="", encoding=encoding) as f:
        writer = csv.writer(f)
        writer.writerow(["topic", "rank", "term", "weight"])
        weights = h / (h.sum(axis=1)[:, None] + 1e-100)
        for i, name in enumerate(topic_names):
            order = np.argsort(weights[i])[::-1][:top_terms]
            for rank, j in enumerate(order, start=1):
                writer.writerow([name, rank, words_of(terms[j]),
                                 round(float(weights[i, j]), rounding)])

    from datetime import date

    model = {
        "kind": MODEL_KIND,
        "format": MODEL_FORMAT,
        "created": date.today().isoformat(),
        "taters": _model_version(),
        # `device` is deliberately absent: where Stanza ran is a runtime choice,
        # not part of the instrument, and recording it would make two otherwise
        # identical models compare as different.
        "text": {"lemmatize": lemmatize, "pos_tagged": pos_tagged,
                 "engine": engine, "tokenizer": tokenizer,
                 "stanza_lang": stanza_lang,
                 "keep_punctuation": keep_punctuation},
        "matrix": {"weighting": weighting, "rounding": matrix_rounding,
                   "terms": terms, "columns": columns,
                   "idf": [vocab[g]["idf"] for g in terms]},
        # no seed: NNDSVD initialization is deterministic, so there is nothing
        # random here to pin down and nothing to forget to record.
        "model": {"n_documents": n_documents, "topics": topic_names,
                  "beta_loss": beta_loss, "iters": iters, "tol": tol,
                  "error_trace": [round(float(x), 4) for x in trace],
                  "components": h.tolist()},
    }
    with atomic_write(model_path, encoding="utf-8") as f:
        json.dump(model, f, indent=1)

    _score_matrix(dtm_csv, out_features_csv, h=h, beta_loss=beta_loss,
                  topic_names=topic_names, rounding=rounding, encoding=encoding,
                  on_progress=on_progress, n_documents=n_documents)
    return out_features_csv

apply_nmf_model

apply_nmf_model(
    *,
    model_json,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    workers=0,
    device="auto",
    on_progress=None,
    out_features_csv=None,
    overwrite_existing=False,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    rounding=4
)

Score a corpus with topics fitted somewhere else.

Takes no vocabulary, tokenizer or weighting settings at all: they come out of the model, because the model is the instrument. That is what makes two studies comparable -- measure the second corpus with the first one's topics, rather than fitting new topics and hoping Topic_3 means the same thing.

Parameters:

Name	Type	Description	Default
model_json	str or Path	A model written by :func:topic_model_nmf, or a library folder holding exactly one.	required
csv_path	Optional[PathLike]	The corpus to score, exactly one way.	None
txt_dir	Optional[PathLike]	The corpus to score, exactly one way.	None
analysis_csv	Optional[PathLike]	The corpus to score, exactly one way.	None
gathered_csv	Optional[PathLike]	The corpus to score, exactly one way.	None
workers	int	Worker processes. 0 picks a sensible number.	0
device	('auto', 'cuda', 'cpu')	Where Stanza runs, if the model was built with it.	"auto"
out_features_csv	str or Path	Defaults to ./features/topic_model_nmf/<gathered filename>.	None
overwrite_existing	bool	If False and the output exists, return it unchanged.	False
encoding	str	Encoding for reading and writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	Gathering options, as in :func:topic_model_nmf.	('text',)
id_cols	Sequence[str]	Gathering options, as in :func:topic_model_nmf.	('text',)
mode	Sequence[str]	Gathering options, as in :func:topic_model_nmf.	('text',)
group_by	Sequence[str]	Gathering options, as in :func:topic_model_nmf.	('text',)
pattern	Sequence[str]	Gathering options, as in :func:topic_model_nmf.	('text',)
rounding	int	Decimal places in the written proportions.	4

Returns:

Type	Description
Path	out_features_csv: text_id, token_count, then one column per topic of the model.

Source code in src\taters\text\topic_model_nmf.py

@records_settings(
    # the model file is the whole instrument -- vocabulary, tokenizer settings
    # and topics -- so it is compared by content. two models with the same name
    # and different topics have to come out as different measurements.
    binding=TEXT_INPUT, grain=TEXT_GRAIN, assets={"model_json": None},
    outputs=("out_features_csv",), bookkeeping=("token_count",))
def apply_nmf_model(
    *,
    model_json: PathLike,

    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    workers: int = 0,
    device: str = "auto",
    on_progress: Optional[Callable[..., None]] = None,

    # ----- Output -----
    out_features_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ====== TXT FOLDER GATHER OPTIONS ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    rounding: int = 4,
) -> Path:
    """
    Score a corpus with topics fitted somewhere else.

    Takes no vocabulary, tokenizer or weighting settings at all: they come out
    of the model, because the model *is* the instrument. That is what makes two
    studies comparable -- measure the second corpus with the first one's topics,
    rather than fitting new topics and hoping Topic_3 means the same thing.

    Parameters
    ----------
    model_json : str or pathlib.Path
        A model written by :func:`topic_model_nmf`, or a library folder holding
        exactly one.
    csv_path, txt_dir, analysis_csv, gathered_csv
        The corpus to score, exactly one way.
    workers : int, default=0
        Worker processes. 0 picks a sensible number.
    device : {"auto", "cuda", "cpu"}, default="auto"
        Where Stanza runs, if the model was built with it.
    out_features_csv : str or pathlib.Path, optional
        Defaults to ``./features/topic_model_nmf/<gathered filename>``.
    overwrite_existing : bool, default=False
        If ``False`` and the output exists, return it unchanged.
    encoding : str, default="utf-8-sig"
        Encoding for reading and writing CSV files.
    text_cols, id_cols, mode, group_by, pattern
        Gathering options, as in :func:`topic_model_nmf`.
    rounding : int, default=4
        Decimal places in the written proportions.

    Returns
    -------
    Path
        ``out_features_csv``: ``text_id``, ``token_count``, then one column per
        topic of the model.
    """
    import numpy as np

    model = _load_model(one_model_path(model_json))
    text_cfg = model["text"]
    matrix_cfg = model["matrix"]
    fit = model["model"]

    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_features_csv is None:
        # `_applied`, so that fitting and applying with the defaults do not
        # write to one file -- and so their matrix folders, derived from this
        # stem, can never be the same one.
        out_features_csv = (Path.cwd() / "features" / "topic_model_nmf"
                            / f"{analysis_ready.stem}_applied{analysis_ready.suffix}")
    out_features_csv = Path(out_features_csv)
    out_features_csv.parent.mkdir(parents=True, exist_ok=True)
    if not overwrite_existing and out_features_csv.is_file():
        print("NMF factor output file already exists; returning existing file.")
        return out_features_csv

    # rebuild the matrix the model's way, not the caller's. the vocabulary is
    # pinned to the model's own term list, so a word the training corpus never
    # saw simply does not count -- which is the honest thing: the model has no
    # topic for it.
    from .build_doc_term_matrix import build_doc_term_matrix

    # the apply gets its own matrix folder, never the fit's: the frozen
    # frequency list written below is deliberately *not* provenance-
    # recorded -- it is the model's vocabulary, not a measurement of this
    # corpus. Sharing one folder meant a later fit found that file, saw no
    # record to disagree with, reused it, and modeled the applied model's
    # vocabulary instead of its own corpus's. Fit on A, apply to B, fit on B,
    # and the second model came out holding A's words, with no error anywhere.
    # the matrix folder is derived from the output's *stem*, not from its
    # folder. a fixed-name sibling (`parent / "matrix"`) is the same path for
    # every step that writes into one folder, and that produced the two worst
    # bugs in this feature: three topic models building over each other's
    # matrix, and an apply leaving a frozen vocabulary where a later fit picked
    # it up and modeled the wrong corpus. Deriving from the stem makes both
    # impossible rather than merely wired-around -- and it is the idiom the
    # rest of the codebase already uses for companion files.
    matrix_dir = out_features_csv.with_name(
        f"{out_features_csv.stem}_matrix")
    matrix_dir.mkdir(parents=True, exist_ok=True)
    dtm_csv = Path(build_doc_term_matrix(
        freq_list_csv=_frozen_freq_list(model, matrix_dir, encoding=encoding),
        analysis_csv=analysis_ready,
        out_features_csv=matrix_dir / "dtm.csv",
        overwrite_existing=True, workers=workers, on_progress=on_progress,
        encoding=encoding, weighting=matrix_cfg["weighting"],
        rounding=int(matrix_cfg["rounding"]),
        vocab_min_freq=0, vocab_min_obs_pct=0, vocab_rule="top_n",
        vocab_top_n=len(matrix_cfg["terms"]), vocab_rank_by="frequency",
        lemmatize=text_cfg["lemmatize"], pos_tagged=text_cfg["pos_tagged"],
        engine=text_cfg["engine"], tokenizer=text_cfg["tokenizer"],
        stanza_lang=text_cfg["stanza_lang"],
        keep_punctuation=bool(text_cfg.get("keep_punctuation", True)),
        device=device))

    h = np.asarray(fit["components"], dtype=np.float64)
    n_documents = sum(1 for _ in Path(dtm_csv).open(encoding=encoding)) - 1
    # the divergence the model was fitted with, not this run's default: the
    # two have different update rules, so scoring a KL model with the
    # least-squares one gives weights the fit never produced.
    _score_matrix(dtm_csv, out_features_csv, h=h,
                  beta_loss=str(fit.get("beta_loss", "frobenius")),
                  topic_names=list(fit["topics"]), rounding=rounding,
                  encoding=encoding, on_progress=on_progress,
                  n_documents=max(n_documents, 0))
    return out_features_csv

taters.text.topic_count_sweep

How many topics? Fit several, score them, and look at the curve.

The one setting a topic model cannot pick for you

Everything else about LDA and NMF has a defensible default. The number of topics does not. Ask for five and you get broad ones ("food", "work"); ask for fifty on the same corpus and you get narrow ones ("breakfast", "restaurant complaints"). Neither is wrong -- they are different questions -- and no statistic can tell you which question you meant to ask.

What this does is narrower and still useful: fit the model at each number you name, score how well each one's topics hang together, and write the curve out with the top words for every fit. Then you read them.

Topic coherence, and what it is not

Coherence asks whether the words that define a topic actually turn up in the same documents. A topic of bread, butter, cheese scores well because those co-occur; a topic of bread, deadline, quarterly scores badly because they do not. That correlates with topics a person would call meaningful, and it is not the same thing -- a model can score beautifully and still carve the corpus somewhere useless.

So this reports and recommends. It does not decide, and the report says so. Anyone who picks the peak of this curve without reading the words has replaced a judgment call with a number that was never meant to carry it.

Both metrics come out of the document-term matrix in a single pass -- see _topics.coherence. C_v, the one most papers quote, is deliberately missing: it needs sliding windows over the raw text, and wanting it is the usual reason people end up adding gensim.

sweep_topic_count

sweep_topic_count(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    workers=0,
    device="auto",
    on_progress=None,
    out_csv=None,
    out_chart_png=None,
    out_report_md=None,
    overwrite_existing=False,
    encoding="utf-8-sig",
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    joiner=" ",
    num_buckets=512,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern=DOCUMENT_PATTERN,
    id_from="stem",
    include_source_path=True,
    ngram_n=1,
    stoplist_paths=None,
    min_freq=5,
    min_obs_pct=0.1,
    min_token_count=10,
    min_npmi=None,
    lemmatize=False,
    pos_tagged=False,
    engine_nlp="nltk",
    tokenizer="potts",
    stanza_lang="en",
    keep_punctuation=False,
    matrix_rounding=4,
    vocab_min_freq=0,
    vocab_min_obs_pct=0,
    vocab_rule="top_n",
    vocab_top_n=2000,
    vocab_rank_by="obs_pct",
    engine="lda",
    k_values="5,10,20,40",
    rule="coherence_exclusivity",
    metric="npmi",
    top_terms=10,
    passes=10,
    seed=42,
    beta_loss="frobenius",
    rounding=4
)

Fit a topic model at several topic counts and score each one's coherence.

Parameters:

Name	Type	Description	Default
csv_path	Optional[PathLike]	The corpus, exactly one way.	None
txt_dir	Optional[PathLike]	The corpus, exactly one way.	None
analysis_csv	Optional[PathLike]	The corpus, exactly one way.	None
gathered_csv	Optional[PathLike]	The corpus, exactly one way.	None
workers	int	Worker processes for gathering and counting. 0 picks a sensible number.	0
device	('auto', 'cuda', 'cpu')	Where Stanza runs, with engine_nlp="stanza".	"auto"
out_csv	str or Path	One row per topic count: the coherence, and the spread across topics. Defaults to ./features/topic_count_sweep/<gathered filename>.	None
out_chart_png	str or Path	The curve, and a short write-up naming the best score and listing the top words of every fit. Written beside out_csv by default.	None
out_report_md	str or Path	The curve, and a short write-up naming the best score and listing the top words of every fit. Written beside out_csv by default.	None
overwrite_existing	bool	If False and the output exists, return it unchanged.	False
encoding	str	Encoding for reading and writing CSV files.	"utf-8-sig"
text_cols	Sequence[str]	Gathering options, as elsewhere.	('text',)
id_cols	Sequence[str]	Gathering options, as elsewhere.	('text',)
mode	Sequence[str]	Gathering options, as elsewhere.	('text',)
group_by	Sequence[str]	Gathering options, as elsewhere.	('text',)
pattern	Sequence[str]	Gathering options, as elsewhere.	('text',)
ngram_n	int	How the vocabulary is counted. Built once and shared by every fit: changing the vocabulary between fits would mean comparing coherence scores computed over different word lists, which compares nothing.	1
stoplist_paths	int	How the vocabulary is counted. Built once and shared by every fit: changing the vocabulary between fits would mean comparing coherence scores computed over different word lists, which compares nothing.	1
min_freq	int	How the vocabulary is counted. Built once and shared by every fit: changing the vocabulary between fits would mean comparing coherence scores computed over different word lists, which compares nothing.	1
min_obs_pct	int	How the vocabulary is counted. Built once and shared by every fit: changing the vocabulary between fits would mean comparing coherence scores computed over different word lists, which compares nothing.	1
min_token_count	int	How the vocabulary is counted. Built once and shared by every fit: changing the vocabulary between fits would mean comparing coherence scores computed over different word lists, which compares nothing.	1
min_npmi	int	How the vocabulary is counted. Built once and shared by every fit: changing the vocabulary between fits would mean comparing coherence scores computed over different word lists, which compares nothing.	1
lemmatize	bool	Tokenizer settings. engine_nlp rather than engine because engine here already means which topic model to fit.	False
pos_tagged	bool	Tokenizer settings. engine_nlp rather than engine because engine here already means which topic model to fit.	False
engine_nlp	bool	Tokenizer settings. engine_nlp rather than engine because engine here already means which topic model to fit.	False
tokenizer	bool	Tokenizer settings. engine_nlp rather than engine because engine here already means which topic model to fit.	False
stanza_lang	bool	Tokenizer settings. engine_nlp rather than engine because engine here already means which topic model to fit.	False
keep_punctuation	bool	Tokenizer settings. engine_nlp rather than engine because engine here already means which topic model to fit.	False
matrix_rounding	int	Decimal places in the shared matrix.	4
vocab_min_freq	float	How the vocabulary is cut from the frequency list.	0
vocab_min_obs_pct	float	How the vocabulary is cut from the frequency list.	0
vocab_rule	float	How the vocabulary is cut from the frequency list.	0
vocab_top_n	float	How the vocabulary is cut from the frequency list.	0
vocab_rank_by	float	How the vocabulary is cut from the frequency list.	0
engine	('lda', 'nmf')	Which model to fit. The matrix follows: counts for LDA, tf-idf for NMF, exactly as when fitting one for real.	"lda"
k_values	str or sequence of int	The topic counts to try, as a list or a comma-separated string.	"5,10,20,40"
metric	('npmi', 'umass')	Which coherence to score with. npmi is normalized to [-1, 1], which is what makes scores comparable between topic counts -- and comparing between topic counts is the entire point here. umass is the older asymmetric measure, for comparison against papers that report it.	"npmi"
top_terms	int	How many words of each topic the coherence is computed over, and how many the report lists. Keep it well below vocab_top_n / n_topics: if the vocabulary cannot give each topic that many distinct words, the lists overlap and coherence starts measuring words from different topics against each other. A warning says so when it happens.	10
passes	int	LDA's settings; ignored for NMF, which needs no seed.	10
seed	int	LDA's settings; ignored for NMF, which needs no seed.	10
beta_loss	('frobenius', 'kullback-leibler')	NMF's divergence; ignored for LDA.	"frobenius"
rounding	int	Decimal places in the written scores.	4

Returns:

Type	Description
Path	out_csv: n_topics, coherence, coherence_sd, worst_topic.

Notes

Coherence is a guide, not a verdict, and this is not an optimizer. Fitting is repeated once per topic count, so a sweep over four counts costs about four fits -- the matrix, which is the slow part on a big corpus, is built once and reused.

Source code in src\taters\text\topic_count_sweep.py

@records_settings(
    binding=TEXT_INPUT, grain=TEXT_GRAIN,
    outputs=("out_csv", "out_chart_png", "out_report_md"))
def sweep_topic_count(
    *,
    # ----- Input source (choose exactly one, or pass analysis_csv directly) -----
    csv_path: Optional[PathLike] = None,
    txt_dir: Optional[PathLike] = None,
    analysis_csv: Optional[PathLike] = None,
    gathered_csv: Optional[PathLike] = None,
    workers: int = 0,
    device: str = "auto",
    on_progress: Optional[Callable[..., None]] = None,

    # ----- Output -----
    out_csv: Optional[PathLike] = None,
    out_chart_png: Optional[PathLike] = None,
    out_report_md: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    encoding: str = "utf-8-sig",

    # ====== CSV GATHER OPTIONS ======
    text_cols: Sequence[str] = ("text",),
    id_cols: Optional[Sequence[str]] = None,
    mode: Literal["concat", "separate"] = "concat",
    group_by: Optional[Sequence[str]] = None,
    delimiter: str = ",",
    joiner: str = " ",
    num_buckets: int = 512,
    max_open_bucket_files: int = 64,
    tmp_root: Optional[PathLike] = None,

    # ====== TXT FOLDER GATHER OPTIONS ======
    recursive: bool = True,
    pattern: str = DOCUMENT_PATTERN,
    id_from: Literal["stem", "name", "path"] = "stem",
    include_source_path: bool = True,

    # ----- the vocabulary, built once and shared by every fit -----
    ngram_n: int = 1,
    stoplist_paths: Optional[Sequence[PathLike]] = None,
    min_freq: int = 5,
    min_obs_pct: float = 0.10,
    min_token_count: int = 10,
    min_npmi: Optional[float] = None,
    lemmatize: bool = False,
    pos_tagged: bool = False,
    engine_nlp: Literal["nltk", "stanza"] = "nltk",
    tokenizer: Literal["potts", "stanza"] = "potts",
    stanza_lang: str = "en",
    keep_punctuation: bool = False,
    matrix_rounding: int = 4,
    vocab_min_freq: float = 0,
    vocab_min_obs_pct: float = 0,
    vocab_rule: Literal["top_n", "min_obs_pct", "min_freq"] = "top_n",
    vocab_top_n: int = 2000,
    vocab_rank_by: Literal["obs_pct", "frequency"] = "obs_pct",

    # ----- the sweep -----
    engine: Literal["lda", "nmf"] = "lda",
    k_values: Union[str, Sequence[int]] = "5,10,20,40",
    rule: Literal["coherence", "coherence_exclusivity"] = "coherence_exclusivity",
    metric: Literal["npmi", "umass"] = "npmi",
    top_terms: int = 10,
    passes: int = 10,
    seed: int = 42,
    beta_loss: Literal["frobenius", "kullback-leibler"] = "frobenius",
    rounding: int = 4,
) -> Path:
    """
    Fit a topic model at several topic counts and score each one's coherence.

    Parameters
    ----------
    csv_path, txt_dir, analysis_csv, gathered_csv
        The corpus, exactly one way.
    workers : int, default=0
        Worker processes for gathering and counting. 0 picks a sensible number.
    device : {"auto", "cuda", "cpu"}, default="auto"
        Where Stanza runs, with ``engine_nlp="stanza"``.
    out_csv : str or pathlib.Path, optional
        One row per topic count: the coherence, and the spread across topics.
        Defaults to ``./features/topic_count_sweep/<gathered filename>``.
    out_chart_png, out_report_md : str or pathlib.Path, optional
        The curve, and a short write-up naming the best score and listing the
        top words of every fit. Written beside ``out_csv`` by default.
    overwrite_existing : bool, default=False
        If ``False`` and the output exists, return it unchanged.
    encoding : str, default="utf-8-sig"
        Encoding for reading and writing CSV files.
    text_cols, id_cols, mode, group_by, pattern
        Gathering options, as elsewhere.
    ngram_n, stoplist_paths, min_freq, min_obs_pct, min_token_count, min_npmi
        How the vocabulary is counted. Built **once** and shared by every fit:
        changing the vocabulary between fits would mean comparing coherence
        scores computed over different word lists, which compares nothing.
    lemmatize, pos_tagged, engine_nlp, tokenizer, stanza_lang, keep_punctuation
        Tokenizer settings. ``engine_nlp`` rather than ``engine`` because
        ``engine`` here already means which topic model to fit.
    matrix_rounding : int, default=4
        Decimal places in the shared matrix.
    vocab_min_freq, vocab_min_obs_pct, vocab_rule, vocab_top_n, vocab_rank_by
        How the vocabulary is cut from the frequency list.
    engine : {"lda", "nmf"}, default="lda"
        Which model to fit. The matrix follows: counts for LDA, tf-idf for NMF,
        exactly as when fitting one for real.
    k_values : str or sequence of int, default="5,10,20,40"
        The topic counts to try, as a list or a comma-separated string.
    metric : {"npmi", "umass"}, default="npmi"
        Which coherence to score with. ``npmi`` is normalized to [-1, 1], which
        is what makes scores comparable **between** topic counts -- and comparing
        between topic counts is the entire point here. ``umass`` is the older
        asymmetric measure, for comparison against papers that report it.
    top_terms : int, default=10
        How many words of each topic the coherence is computed over, and how
        many the report lists. Keep it well below ``vocab_top_n / n_topics``:
        if the vocabulary cannot give each topic that many distinct words, the
        lists overlap and coherence starts measuring words from different
        topics against each other. A warning says so when it happens.
    passes, seed : int
        LDA's settings; ignored for NMF, which needs no seed.
    beta_loss : {"frobenius", "kullback-leibler"}
        NMF's divergence; ignored for LDA.
    rounding : int, default=4
        Decimal places in the written scores.

    Returns
    -------
    Path
        ``out_csv``: ``n_topics``, ``coherence``, ``coherence_sd``, ``worst_topic``.

    Notes
    -----
    Coherence is a guide, not a verdict, and this is not an optimizer. Fitting
    is repeated once per topic count, so a sweep over four counts costs about
    four fits -- the matrix, which is the slow part on a big corpus, is built
    once and reused.
    """
    if engine not in ENGINES:
        raise ValueError(f"unknown engine {engine!r}; have {ENGINES}")
    if metric not in _topics.COHERENCE_METRICS:
        raise ValueError(
            f"unknown coherence metric {metric!r}; have "
            f"{_topics.COHERENCE_METRICS}. c_v is not available -- it needs "
            "sliding windows over the raw text rather than the matrix.")
    counts = _parse_counts(k_values)


    analysis_ready = resolve_analysis_ready(
        csv_path=csv_path, txt_dir=txt_dir, analysis_csv=analysis_csv,
        gathered_csv=gathered_csv, text_cols=text_cols, id_cols=id_cols,
        mode=mode, group_by=group_by, delimiter=delimiter, encoding=encoding,
        joiner=joiner, num_buckets=num_buckets,
        max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
        recursive=recursive, pattern=pattern, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers)

    if out_csv is None:
        out_csv = (Path.cwd() / "features" / "topic_count_sweep"
                   / analysis_ready.name)
    out_csv = Path(out_csv)
    out_csv.parent.mkdir(parents=True, exist_ok=True)
    stem = out_csv.stem
    chart_path = Path(out_chart_png) if out_chart_png else \
        out_csv.with_name(f"{stem}_coherence.png")
    report_path = Path(out_report_md) if out_report_md else \
        out_csv.with_name(f"{stem}_report.md")

    if not overwrite_existing and out_csv.is_file():
        print("Topic-count sweep output already exists; returning existing file.")
        return out_csv

    # one matrix, every fit. rebuilding it per topic count would mean scoring
    # coherence over a different vocabulary each time, and those numbers are
    # not comparable -- which would defeat the only thing a sweep is for.
    weighting = "count" if engine == "lda" else "tfidf"
    _freq, dtm_csv, terms, _columns, _vocab = _topics.build_matrix(
    # the matrix folder is derived from the output's *stem*, not from its
    # folder. a fixed-name sibling (`parent / "matrix"`) is the same path for
    # every step that writes into one folder, and that produced the two worst
    # bugs in this feature: three topic models building over each other's
    # matrix, and an apply leaving a frozen vocabulary where a later fit picked
    # it up and modeled the wrong corpus. Deriving from the stem makes both
    # impossible rather than merely wired-around -- and it is the idiom the
    # rest of the codebase already uses for companion files.
        analysis_ready=analysis_ready,
        out_dir=out_csv.with_name(f"{stem}_matrix"),
        weighting=weighting, matrix_rounding=matrix_rounding,
        text_settings=dict(lemmatize=lemmatize, pos_tagged=pos_tagged,
                           engine=engine_nlp, tokenizer=tokenizer,
                           stanza_lang=stanza_lang,
                           keep_punctuation=keep_punctuation, device=device),
        vocab_settings=dict(vocab_min_freq=vocab_min_freq,
                            vocab_min_obs_pct=vocab_min_obs_pct,
                            vocab_rule=vocab_rule, vocab_top_n=vocab_top_n,
                            vocab_rank_by=vocab_rank_by),
        ngram_settings=dict(ngram_n=ngram_n, stoplist_paths=stoplist_paths,
                            min_freq=min_freq, min_obs_pct=min_obs_pct,
                            min_token_count=min_token_count, min_npmi=min_npmi),
        overwrite_existing=overwrite_existing, workers=workers,
        on_progress=on_progress, encoding=encoding)

    def batches():
        return _topics.stream_counts(dtm_csv, encoding=encoding, skip_cols=2)

    # NMF needs the whole matrix at once and cannot stream, so it is read here
    # rather than once per candidate count.
    matrix = None
    if engine == "nmf":
        n_docs_est = sum(len(block) for block in batches())
        wanted = _topics.nmf_memory_gb(n_docs_est, len(terms), max(counts))
        if wanted >= 1.0:
            import warnings
            warnings.warn(
                f"NMF holds the whole matrix in memory: about {wanted:.1f} GB "
                f"for {n_docs_est:,} documents by {len(terms):,} terms. Reduce "
                "`vocab_top_n` if that is more than this machine has.")
        matrix = _topics.read_matrix(dtm_csv, encoding=encoding, skip_cols=2)

    def fit_at(k):
        if engine == "lda":
            return _topics.fit_lda(batches, len(terms), k, passes=passes,
                                   seed=seed)[0]
        return _topics.fit_nmf(matrix, k, beta_loss=beta_loss)[1]

    # everything below here -- which counts this corpus can carry, the
    # co-occurrence pass, the scoring, the table, the charts and the write-up
    # -- is the same machinery the three topic models use when their own count
    # is left at 0. This function is the standalone door onto it, for looking
    # at the curve without committing to a model.
    _best_k, _components = _topics.select_k(
        k_values=counts, fit=fit_at, terms=terms, batches=batches,
        engine=engine, rule=rule, metric=metric, top_terms=top_terms,
        rounding=rounding, out_stem=out_csv.with_suffix(""),
        chart=chart_path, report=report_path,
        encoding=encoding, on_progress=on_progress)
    return out_csv