`a953bb4`

fix(critical): chatbot detection was checking filtered content

**The Bug:**
Chatbot detection was running AFTER safeguards filtered code blocks from content.
By the time _contains_unexecuted_code() ran, the code blocks were already gone,
so detection always failed!

**The Fix:**
Check response_content (original) instead of content (filtered).
Now detects numbered steps, code blocks, and tutorial patterns correctly.

**Why This Matters:**
Without this fix, even large models like Mixtral would slip into chatbot mode
and never get corrected because the detection was blind.

**Added:**
- Debug logging to trace detection decisions
- Check original content before filtering
- Visibility into what content is being evaluated

Authored by

espadonne 4 months ago

SHA: a953bb4ab3b8d865dcf1df1376f1c8a986e92c1e
Parents: a97c3af
Tree: a22af7c

1 changed file

Status	File	+	-
M	`src/loader/agent/loop.py`	17	1

src/loader/agent/loop.pymodified

                  continue
              # No tool calls - check if model is describing instead of acting
 -            if self._contains_unexecuted_code(content) and iterations < self.config.max_iterations - 1:
 +            # IMPORTANT: Check ORIGINAL content before safeguards filtered it!
 +            # Debug log
 +            try:
 +                has_unexecuted = self._contains_unexecuted_code(response_content)
 +                with open("/tmp/loader_debug.log", "a") as f:
 +                    f.write(f"[chatbot-check] iterations={iterations}, has_unexecuted={has_unexecuted}\n")
 +                    f.write(f"[chatbot-check] response_content (first 200): {response_content[:200]}\n")
 +                    f.write(f"[chatbot-check] filtered content (first 200): {content[:200]}\n")
 +            except Exception:
 +                pass
++
 +            if self._contains_unexecuted_code(response_content) and iterations < self.config.max_iterations - 1:
                  # Model outputted code blocks without using tools - nudge it
 +                try:
 +                    with open("/tmp/loader_debug.log", "a") as f:
 +                        f.write(f"[chatbot-check] TRIGGERING chatbot recovery\n")
 +                except Exception:
 +                    pass
                  await emit(AgentEvent(
                      type="error",
                      content="⚠ Chatbot mode detected - steering agent to use tools instead of giving instructions",