`a97c3af`

fix: improve chatbot mode detection and recovery

Enhances detection of tutorial-style responses to catch small models that give
numbered instructions instead of using tools.

**New Detection Patterns:**
- Numbered lists (1., 2., 3. Open..., Create..., Navigate...)
- Sequenced steps (First..., Next..., Then...)
- Tutorial starters ('Open your terminal...', 'Navigate to...')
- How-to preambles ('Here's how you can...')

**Stronger Steering:**
- More explicit error message when chatbot mode is detected
- Clear examples of what NOT to do vs what TO do
- User-visible warning when auto-correction is triggered

This helps smaller models like llama3.2:3b stay in agent mode instead
of reverting to chatbot/tutorial behavior.

Authored by

espadonne 4 months ago

SHA: a97c3af7ade0f5d5341357f7127a66f165d5f912
Parents: ee05471
Tree: 9cad6c4

1 changed file

Status	File	+	-
M	`src/loader/agent/loop.py`	28	3

src/loader/agent/loop.pymodified

              # No tool calls - check if model is describing instead of acting
              if self._contains_unexecuted_code(content) and iterations < self.config.max_iterations - 1:
                  # Model outputted code blocks without using tools - nudge it
 +                await emit(AgentEvent(
 +                    type="error",
 +                    content="⚠ Chatbot mode detected - steering agent to use tools instead of giving instructions",
 +                ))
                  self.messages.append(Message(
                      role=Role.ASSISTANT,
                      content=response_content,
                  ))
                  self.messages.append(Message(
                      role=Role.USER,
 -                    content="STOP. Do not show me code to copy. USE YOUR TOOLS to execute the actions. "
 -                            "Call the bash tool to run commands. Call the write tool to create files. "
 -                            "Execute the task NOW using tool calls.",
 +                    content="CRITICAL ERROR: You are giving me instructions to copy instead of EXECUTING the task.\n\n"
 +                            "DO NOT write:\n"
 +                            "- Numbered steps (1., 2., 3.)\n"
 +                            "- Instructions like 'Open your terminal...'\n"
 +                            "- Code blocks for me to copy\n"
 +                            "- 'You can run...', 'Create this file...'\n\n"
 +                            "INSTEAD: Use your bash and write tools RIGHT NOW to execute the task.\n"
 +                            "Example: [call write tool], [call bash tool]\n"
 +                            "DO IT NOW - don't describe it.",
                  ))
                  continue
              'execute this', 'paste this',
+         ]
 +        # Tutorial/instruction patterns
 +        tutorial_patterns = [
 +            r'^\s*\d+\.\s+(open|create|navigate|run|execute|make)',  # Numbered instructions
 +            r'(first|second|third|next|then),?\s+(open|create|navigate)',  # Sequenced steps
 +            r'open your (terminal|command|shell)',  # Tutorial starter
 +            r'navigate to (the|your|~/)',  # Navigation instruction
 +            r'here\'s how you can (quickly|easily)?',  # How-to preamble
 +            r'you can (start by|begin by|follow these)',  # Tutorial start
 +        ]
++
          content_lower = content.lower()
 +        # Check for tutorial patterns
 +        for pattern in tutorial_patterns:
 +            if re.search(pattern, content_lower, re.MULTILINE | re.IGNORECASE):
 +                return True
++
          # If chatbot phrases present with code blocks, it's describing not doing
          for phrase in chatbot_phrases:
              if phrase in content_lower: