`9e90646`

templates/gallery: tool-use-example.dlm starter for sway tool_use_fidelity probe

Authored by mfwolffe <wolffemf@dukes.jmu.edu> 2 weeks ago

SHA: 9e90646e1ac6a30117ecacd8b43a83a793f34116
Parents: f138f4c
Tree: 42c8959

2 changed files

Status	File	+	-
A	`src/dlm/templates/gallery/tool-use-example.dlm`	126	0
A	`src/dlm/templates/gallery/tool-use-example.meta.yaml`	25	0

src/dlm/templates/gallery/tool-use-example.dlmadded

++---
++dlm_id: 01KQ7XKG7AZ1VGFD2836YHK01V
++dlm_version: 1
++base_model: qwen2.5-coder-1.5b
++system_prompt: |
++  You are a tool-using assistant. When the user asks for an action that
++  matches one of your available tools, respond with ONLY a single JSON
++  object of the form {"name": "<tool_name>", "arguments": { ... }}.
++  Do not add prose, code fences, or explanations — just the JSON.
++training:
++  adapter: lora
++  lora_r: 16
++  lora_alpha: 32
++  lora_dropout: 0.05
++  sequence_len: 1024
++  learning_rate: 2e-4
++  num_epochs: 3
++  seed: 42
++export:
++  default_quant: Q4_K_M
++  default_temperature: 0.0
++---
++
++# Tool-using assistant starter
++
++Trains an adapter that always answers tool requests with a strict JSON
++function-call object — never prose, never markdown. Pair the resulting
++adapter with sway's `tool_use_fidelity` probe (see the linked sway
++spec at the end) to measure JSON-schema validity, argument drift, and
++tool-name hallucination after training.
++
++The Q&A blocks below cover the eight tool shapes sway's bundled
++`tool_use_cases.yaml` fixture exercises: search, file I/O, code
++execution, shell, HTTP, DB, calendar, and calculator. Replace or
++extend with the real tool surface your application exposes.
++
++::instruction::
++### Q
++Search the web for the largest known prime number. Use the `search_web` tool.
++
++### A
++{"name": "search_web", "arguments": {"query": "largest known prime number"}}
++
++### Q
++Look up recent earthquakes in Japan. Use `search_web` with `max_results=5`.
++
++### A
++{"name": "search_web", "arguments": {"query": "recent earthquakes Japan", "max_results": 5}}
++
++### Q
++Read the contents of /etc/hosts. Use the `read_file` tool.
++
++### A
++{"name": "read_file", "arguments": {"path": "/etc/hosts"}}
++
++### Q
++Read /var/log/syslog as UTF-8. Use the `read_file` tool.
++
++### A
++{"name": "read_file", "arguments": {"path": "/var/log/syslog", "encoding": "utf-8"}}
++
++### Q
++Compute the factorial of 7 using Python. Use the `run_python` tool.
++
++### A
++{"name": "run_python", "arguments": {"code": "import math; print(math.factorial(7))"}}
++
++### Q
++List files in /tmp using a shell command. Use the `run_shell` tool.
++
++### A
++{"name": "run_shell", "arguments": {"command": "ls -la /tmp"}}
++
++### Q
++GET https://example.com/health. Use the `http_fetch` tool.
++
++### A
++{"name": "http_fetch", "arguments": {"url": "https://example.com/health"}}
++
++### Q
++GET https://api.example.com/v1/users with an Authorization header. Use the `http_fetch` tool.
++
++### A
++{"name": "http_fetch", "arguments": {"url": "https://api.example.com/v1/users", "headers": {"Authorization": "Bearer TOKEN"}}}
++
++### Q
++Fetch the user with id=5 from the database. Use the `db_query` tool.
++
++### A
++{"name": "db_query", "arguments": {"sql": "SELECT * FROM users WHERE id = ?", "params": [5]}}
++
++### Q
++Add a calendar event for a sync at 2pm tomorrow titled "Sync". Use `calendar_create_event`.
++
++### A
++{"name": "calendar_create_event", "arguments": {"title": "Sync", "start_iso": "2026-04-28T14:00:00", "duration_minutes": 30}}
++
++### Q
++Compute 137 * 42 with the `calculator` tool.
++
++### A
++{"name": "calculator", "arguments": {"expression": "137 * 42"}}
++
++### Q
++Compute the area of a circle with radius 5 using the `calculator` tool.
++
++### A
++{"name": "calculator", "arguments": {"expression": "3.14159 * 5 * 5"}}
++::end::
++
++# Sway evaluation
++
++After `dlm train` produces an adapter, evaluate tool-call preservation
++with sway:
++
++```bash
++# install dlm with the sway extra; sway resolves the adapter from the .dlm
++pip install 'dlm[sway]'
++sway autogen tool-use-example.dlm -o sway.yaml
++sway run sway.yaml
++```
++
++The autogenerated suite includes `tool_use_fidelity` cases derived
++from this document's INSTRUCTION blocks. Look for the probe row in
++the sway report — a healthy adapter scores `validity_delta ≥ 0` and
++`hallucination_rate < 10%` against the tools above.

src/dlm/templates/gallery/tool-use-example.meta.yamladded

++name: tool-use-example
++title: Tool-using assistant (JSON function calls)
++domain_tags:
++  - tool-use
++  - agent
++  - function-calling
++  - json
++recommended_base: qwen2.5-coder-1.5b
++expected_steps: 600
++expected_duration:
++  cuda-sm80+: "~4 min"
++  mps: "~12 min"
++  cpu: "~90 min"
++summary: |
++  A compact starter for adapters that emit OpenAI-shape JSON tool
++  calls. Twelve seeded Q&A pairs span the tool shapes sway's
++  `tool_use_fidelity` probe expects: search, file I/O, Python /
++  shell exec, HTTP fetch, DB query, calendar event, calculator.
++  Pair with `sway autogen` + `sway run` to score JSON-schema
++  validity, argument drift, and tool-name hallucination after
++  training.
++sample_prompts:
++  - "Use the search_web tool to find recent papers about diffusion models."
++  - "Read /etc/hosts. Use the read_file tool."
++  - "Compute 8 factorial. Use the calculator tool."