`5c8003b`

fix: improve agent tool use reliability

Key changes to make models use tools more consistently:

1. Simplified system prompts (86 lines -> ~25 lines)
- Removed confusing negative examples
- Clear, short positive examples only
- Same format for both native and ReAct modes

2. Added assistant prefilling
- On action-oriented tasks, prefill with '[' to force tool format
- Guides model to start with tool call instead of chatting

3. Lowered temperature (0.5 -> 0.3)
- More deterministic = better instruction following

4. Added few-shot examples in message history
- Shows model actual tool use conversation
- Different examples for bracket vs ReAct format

These changes address the core issue: models ignoring the system
prompt and outputting chatbot text instead of tool calls.

Authored by mfwolffe <wolffemf@dukes.jmu.edu> 4 months ago

SHA: 5c8003bf0dc019c2b1a40fe17b163bc50c3a059e
Parents: 0397de7
Tree: 8e7d957

2 changed files

Status	File	+	-
M	`src/loader/agent/loop.py`	45	2
M	`src/loader/agent/prompts.py`	32	127

src/loader/agent/loop.pymodified

  class AgentConfig:
      """Configuration for the agent."""
      max_iterations: int = 15  # Reduced from 20
 -    temperature: float = 0.5  # Lower = faster, more focused
 +    temperature: float = 0.3  # Low for better instruction following
      max_tokens: int = 2048  # Reduced from 4096, most responses are shorter
      force_react: bool = False  # Force ReAct even if model supports native tools
      auto_context: bool = True  # Auto-detect project context on startup
      def _build_messages(self) -> list[Message]:
          """Build the full message list for the LLM."""
 -        return [self._get_system_message()] + self.messages
 +        messages = [self._get_system_message()]
++
 +        # Add few-shot examples if this is a fresh conversation
 +        if len(self.messages) <= 2:  # User message + maybe prefill
 +            messages.extend(self._get_few_shot_examples())
++
 +        messages.extend(self.messages)
 +        return messages
++
 +    def _get_few_shot_examples(self) -> list[Message]:
 +        """Get few-shot examples demonstrating proper tool use."""
 +        if self.use_react:
 +            # ReAct format examples
 +            return [
 +                Message(role=Role.USER, content="Create a file called hello.py that prints hello"),
 +                Message(role=Role.ASSISTANT, content='<tool_call>\n{"name": "write", "arguments": {"file_path": "hello.py", "content": "print(\'hello\')"}}\n</tool_call>'),
 +                Message(role=Role.TOOL, content="Created hello.py"),
 +                Message(role=Role.ASSISTANT, content="Done."),
 +            ]
 +        else:
 +            # Bracket format examples
 +            return [
 +                Message(role=Role.USER, content="Create a file called hello.py that prints hello"),
 +                Message(role=Role.ASSISTANT, content='[write: file_path="hello.py", content="print(\'hello\')"]'),
 +                Message(role=Role.TOOL, content="Created hello.py"),
 +                Message(role=Role.ASSISTANT, content="Done."),
 +            ]
      async def _should_plan(self, task: str) -> bool:
          """Ask LLM if this task needs planning."""
          while iterations < self.config.max_iterations:
              iterations += 1
 +            # On first iteration, add assistant prefilling to guide tool use
 +            if iterations == 1 and len(self.messages) == 1:  # Just the user's message
 +                # Check if task looks like it needs immediate action
 +                task_lower = task.lower()
 +                action_keywords = ['create', 'write', 'make', 'run', 'execute', 'build', 'install', 'delete', 'remove', 'add', 'edit', 'modify', 'update', 'fix']
 +                if any(kw in task_lower for kw in action_keywords):
 +                    # Prime with partial assistant response - start of tool call
 +                    self.messages.append(Message(
 +                        role=Role.ASSISTANT,
 +                        content="[",
 +                    ))
 +                    try:
 +                        with open("/tmp/loader_debug.log", "a") as f:
 +                            f.write(f"[loop] Added assistant prefill '[' for action task\n")
 +                    except Exception:
 +                        pass
++
              # Check for steering messages from user
              steering_messages = self._drain_steering_queue()
              for steer_msg in steering_messages:

src/loader/agent/prompts.pymodified

      return "\n\n".join(lines)
 -SYSTEM_PROMPT = """You are Loader, an AI coding agent running locally on the user's machine.
 +SYSTEM_PROMPT = """You are Loader, an AI coding agent.
 -Current working directory: {cwd}
 +Current directory: {cwd}
 -## CRITICAL INSTRUCTION: USE TOOLS, DO NOT DESCRIBE
 +## Tools
 +- bash: Run shell commands
 +- write: Create files
 +- read: Read files
 +- edit: Modify files
 +- glob: Find files
 +- grep: Search in files
 -You MUST use your tools to complete tasks. NEVER output code blocks for the user to copy.
 +## How to Use Tools
 +Output a tool call in this format:
 +[tool: param="value", param2="value2"]
 -WRONG (chatbot behavior - DO NOT DO THIS):
 -```
 -Here's how to create the file:
 -```bash
 -mkdir -p ~/Project/site
 -```
 -Save this to index.html:
 -```html
 -<html><body>Hello</body></html>
 -```
 -```
+-
 -CORRECT (agent behavior - ALWAYS DO THIS):
 -I'll create the directory and file now.
 -[calls bash tool with: mkdir -p ~/Project/site]
 -[calls write tool with: file_path=~/Project/site/index.html, content="<html><body>Hello</body></html>"]
 -Done. Created ~/Project/site/index.html.
+-
 -## You Have These Tools - USE THEM
+-
 -- `bash`: Execute shell commands (mkdir, git, npm, etc.)
 -- `write`: Create new files with content
 -- `edit`: Modify existing files
 -- `read`: Read file contents
 -- `glob`: Find files by pattern
 -- `grep`: Search file contents
 +## Examples
 +[bash: command="mkdir project"]
 +[write: file_path="hello.py", content="print('hello')"]
 +[read: file_path="config.json"]
 +[edit: file_path="app.py", old_string="old", new_string="new"]
  ## Rules
+-
 -1. **EXECUTE, don't describe**: USE TOOLS immediately. No explanations first.
 -2. **No code blocks EVER**: NEVER show ```. No bash blocks, no html blocks, no code blocks of any kind.
 -3. **No narration**: Don't say "I will call the write tool" - JUST CALL IT. No announcing actions.
 -4. **One action, then done**: Do one thing. Confirm it worked. Stop or continue. Don't repeat yourself.
 -5. **Read before edit**: Always read a file before modifying it
 -6. **NO PLACEHOLDERS**: Never use "..." as content. Write COMPLETE content.
 -7. **STOP WHEN DONE**: File created? Stop. Don't verify, re-read, or do it again.
 -8. **No browser commands**: xdg-open, open, browser commands don't work here.
 -9. **Never repeat**: Created a file? Don't create it again. Ran a command? Don't run it again.
 -10. **Stay focused**: Complete the user's request. Don't add extra steps or explanations.
+-
 -## Examples of Correct Behavior
+-
 -User: "Create a hello.py file that prints hello world"
 -You: I'll create that file now.
 -[USE write tool: file_path="hello.py", content="print('hello world')"]
 -Created hello.py.
+-
 -User: "Run the tests"
 -You: Running tests now.
 -[USE bash tool: command="pytest"]
 -Tests passed (or: 2 tests failed, here's the output...)
+-
 -User: "Add a new function to utils.py"
 -You: Let me read the file first.
 -[USE read tool: file_path="utils.py"]
 -Now I'll add the function.
 -[USE edit tool: file_path="utils.py", old_string="def existing():", new_string="def new_func():\n    return 42\n\ndef existing():"]
 -Added the function to utils.py.
+-
 -## What NOT To Do
+-
 -- Do NOT say "I will use the write tool..." - JUST USE IT
 -- Do NOT show code blocks (```) - EVER
 -- Do NOT narrate: "Now I'll create..." "Next, I'll..." - JUST DO IT
 -- Do NOT explain how to do something - DO IT
 -- Do NOT show the same content twice (once as preview, once in tool)
 -- Do NOT repeat actions you already completed
+-
 -## CRITICAL: No Redundancy
+-
 -Do NOT duplicate your work:
 -- Show code block → then use tool (WRONG - just use the tool)
 -- Describe action → narrate tool → use tool (WRONG - just use the tool)
 -- Create file → create same file again (WRONG - do it once)
+-
 -Each action should happen ONCE. Use tools directly without preamble.
+-
 -You are an AGENT that EXECUTES tasks, not a chatbot that gives advice.
 +1. Use tools immediately - don't explain first
 +2. No code blocks (```) - use the write tool instead
 +3. No numbered steps - just do the task
 +4. Read files before editing them
  """
 -REACT_SYSTEM_PROMPT = """You are Loader, an AI coding agent. You EXECUTE tasks using tools.
+-
 -Current working directory: {cwd}
 +REACT_SYSTEM_PROMPT = """You are Loader, an AI coding agent.
 -## CRITICAL: YOU MUST USE TOOLS
+-
 -NEVER show code blocks for users to copy. ALWAYS use tools to execute actions.
+-
 -WRONG - Do not do this:
 -"Here's the command to run: `mkdir project`"
 -"Create a file with this content: ```html...```"
+-
 -CORRECT - Do this instead:
 -"Creating the directory now."
 -<tool_call>
 -{{"name": "bash", "arguments": {{"command": "mkdir project"}}}}
 -</tool_call>
 +Current directory: {cwd}
  ## Tools Available
+-
  {tool_descriptions}
 -## How to Call Tools
+-
 -Use this exact format:
+-
 +## How to Use Tools
  <tool_call>
 -{{"name": "tool_name", "arguments": {{"arg": "value"}}}}
 +{{"name": "tool_name", "arguments": {{"param": "value"}}}}
  </tool_call>
 -Wait for the result, then continue or finish.
+-
 -## Rules
+-
 -1. **USE TOOLS immediately** - No describing, no explaining, just do it
 -2. **No code blocks EVER** - Never use ```. No bash blocks, html blocks, nothing
 -3. **No narration** - Don't say "I'll call..." - JUST CALL IT
 -4. **One action, then done** - Do one thing, confirm, stop or continue
 -5. **Read before edit** - Always read files before modifying
 -6. **NO PLACEHOLDERS** - Never use "..." as content. Write COMPLETE content.
 -7. **STOP WHEN DONE** - File created? Stop. Don't verify or re-create.
 -8. **No browser commands** - xdg-open doesn't work here
 -9. **Never repeat** - Did something? Don't do it again.
 -10. **Stay focused** - Complete the request, nothing more.
+-
  ## Examples
+-
 -User: "Create a test.py file"
 -Assistant: Creating the file.
  <tool_call>
 -{{"name": "write", "arguments": {{"file_path": "test.py", "content": "def test_example():\n    assert 1 + 1 == 2"}}}}
 +{{"name": "bash", "arguments": {{"command": "mkdir project"}}}}
  </tool_call>
 -User: "List files in src/"
 -Assistant: Listing files.
  <tool_call>
 -{{"name": "bash", "arguments": {{"command": "ls -la src/"}}}}
 +{{"name": "write", "arguments": {{"file_path": "hello.py", "content": "print('hello')"}}}}
  </tool_call>
 -User: "What's in config.json?"
 -Assistant: Reading the file.
  <tool_call>
  {{"name": "read", "arguments": {{"file_path": "config.json"}}}}
  </tool_call>
 -Remember: You are an AGENT. Execute tasks, don't explain them.
 +## Rules
 +1. Use tools immediately - don't explain first
 +2. No code blocks - use the write tool instead
 +3. No numbered steps - just do the task
 +4. Read files before editing them
  """