tenseleyflow/documentlanguagemodel / 15c7016

     Pragma'd from unit coverage — exercises class-named model load +

     `AutoProcessor.from_pretrained` over real HF weights. Covered by

     """

     if spec.modality != "audio-language":

         raise ValueError(

 """`InferenceBackend` Protocol shared by PyTorch + MLX paths.

 Backends are stateful: `load()` resolves the adapter, loads weights,

 and stashes the live model on `self`; `generate()` is called repeatedly

 against that loaded state; `unload()` releases memory. Pooling /

 """

 from __future__ import annotations

 - `num_beams=1`

 - `temperature=0.0` (technically moot when do_sample=False, but

   some HF code paths still read it — belt and braces)

 When the caller passes `temperature > 0`, we flip `do_sample=True`

 automatically — otherwise a non-zero temperature is silently ignored

     """Render `prompt`, run generation, decode response-only tokens.

     Pragma'd from unit coverage because it calls `model.generate`.

     """

     import torch

   fp16 residual on top of a fp16 base.

 The tokenizer is loaded from the **adapter directory**, not the

 tokenizer state (including `<|pad|>` additions) into the adapter dir

 Heavy imports are deferred; the orchestration logic that picks args,

 paths, and dtypes is unit-testable without HF.

     Pragma'd from unit coverage because it calls `AutoModelForCausalLM.from_pretrained`

     and `PeftModel.from_pretrained`, which each need ~5 seconds and a

     `adapter_name`, when provided, targets the named multi-adapter

     layout (`adapter/<name>/current.txt`). When `None`, uses the flat

     model.eval()

     # Tokenizer from the adapter dir — source of truth after any

     tokenizer = AutoTokenizer.from_pretrained(str(adapter_path))

     return LoadedInference(

 The problem

 -----------

 The solution

 ------------

 `pinned_versions.json` fallback) to learn

 whether QLoRA was in play, cross-references with the current `Capabilities`,

 and emits:

     Decision tree:

     - CUDA host + bnb installed + QLoRA-trained → 4-bit load, no dequant.

     - CUDA host, QLoRA-trained, but bnb missing → dequantize to fp16.

     - Non-QLoRA adapter → load at the host's best precision (bf16 on

       capable CUDA, else fp16).

     """

             attn_implementation="sdpa",

             reason=(

                 f"QLoRA adapter on {backend} host; dequantizing to fp16 "

             ),

         )

     return InferencePlan(