tenseleyflow/documentlanguagemodel / 07260df

 """Dataset assembly — turn parsed `.dlm` sections into a ready-to-train dataset.

 """

 from __future__ import annotations

 class AudioSampleRateMismatch(DataError):  # noqa: N818 — `*Mismatch` mirrors other DataError subclasses

     """Audio file sample rate doesn't match the base's pinned value.

     """

 """End-to-end: parsed `.dlm` sections → (train_ds, val_ds).

 1. Flattens `sections` to dict rows via `sections_to_rows`.

 3. Splits into train / val via the deterministic splitter.

 The split is keyed on each row's `_dlm_section_id` + sub-index, so

 """

 from __future__ import annotations

     """Build a (train, val) `Dataset` pair from parsed `.dlm` sections.

     `seed` is required (not defaulted) so the split is always traceable

     `weights`, when non-empty, expands rows by `(tag_key, tag_value)`

     multipliers before the train/val split — integer factors duplicate

 - neither → `DataFormatError`.

 PREFERENCE rows (`prompt`/`chosen`/`rejected`) are NOT formatted here —

 fails loudly at format time rather than producing silently-wrong data.

 """

 each triple. Missing, duplicated, or reordered headers raise

 `PreferenceParseError`. Empty field bodies are errors — DPO on empty

 text is never intentional.

 """

 from __future__ import annotations

 """Turn `doc.sections.Section` objects into ready-to-train dict rows.

 | Section type | Row shape |

 |---|---|

 Callers that leave `blob_store=None` with media sections in the

 input raise `ValueError` — the row shape isn't viable without the

 actual bytes. Audio rows hold only the path + sha, not the decoded

 Every row carries `_dlm_section_id` so `splitter.split()` can key

 deterministically on (seed, section_id) rather than row index. This is

    token, or labels get corrupted by mid-sequence EOS masking.

    Fallback order: `unk_token` → else add `<|pad|>` as a new special

    token (which grows the vocab and sets `tokenizer_grew=True` for

 2. **chat_template must be present.** Without it, SFTTrainer can't

    render `messages`-shaped rows. We surface a typed

    `TokenizerBringupError` rather than letting SFT fail deep inside

     """Result of `prepare_tokenizer`.

     `tokenizer_grew=True` means a new `<|pad|>` token was added to the

     """

     tokenizer: PreTrainedTokenizerBase

         return False

     # Last resort: add a new pad token. This grows the vocab, which

     tok.add_special_tokens({"pad_token": _PAD_TOKEN_LITERAL})

     return True

 Two functions:

   added-token set changed. Works for any `PreTrainedTokenizerBase`

   (BPE or SentencePiece family).

 - `modules_to_save_for_growth(grew)` — `["embed_tokens", "lm_head"]`

   LoRA config. Per pitfall #4, without the modules_to_save entry the

   new embedding row's output is undefined.

     """True iff `final` has a larger vocab or different added-token set than `base`.

     `vocab_size` comparison catches the `add_special_tokens` path used by

     catches cases where an added token was *replaced* with a same-count

     variant (vocab size unchanged but contents differ) — rare but

     possible when users manually mutate the tokenizer between runs.

 Keyed on `(blob_sha, processor_sha, target_size)` — a blob-bytes

 change, a processor upgrade, or a resize-policy bump each invalidate

 Layout: `<vl-cache>/<blob_sha[:2]>/<blob_sha>.<proc_sha[:12]>.<h>x<w>.npz`.

 Contents: single numpy array stored under the key `pixel_values`.

 Determinism: the keep/extra-copy decision is a hash of

 `(seed, section_id, fractional_index)`. Same seed + same corpus →

 """

 from __future__ import annotations

     An empty `weights` map is a no-op (returns a shallow copy of

     `rows`). Section-ID preservation means the replay corpus still

     tracks per-row identity — the N copies of a repeated row share

     """

     if not weights:

         return list(rows)