Your agent needs a confidence score (or it'll ship broken work with perfect certainty)

Your agent just spent 20 minutes confidently explaining why your API is broken. Except the API is fine — it was making requests to the wrong endpoint the entire time.

This happens because most agents operate with binary confidence: they either "know" something or they don't. There's no middle ground where they say "I'm 60% sure this is right, but let me double-check."

The fix is a confidence scoring system that makes your agent pause when it's unsure.

Here's the pattern that works:

Add this to your agent's system prompt:

Before taking any action that could break something or waste time:

1. Rate your confidence (1-10) in your understanding
2. If confidence < 7: explicitly state assumptions and ask for verification
3. If confidence < 5: stop and request clarification
4. Always show your confidence score in brackets [confidence: 8/10]

But scoring alone isn't enough. Your agent needs calibration — it needs to learn when its confidence scores actually match reality.

I track this by logging every confidence score alongside the actual outcome:

# confidence_log.md

## 2024-12-19
- [confidence: 9/10] "API endpoint is /users/create" → WRONG (was /api/v1/users)
- [confidence: 6/10] "Database needs migration" → CORRECT
- [confidence: 8/10] "Bug is in the auth middleware" → CORRECT

After a week, I review the log. If my agent is scoring 9/10 but getting things wrong 30% of the time, I adjust the calibration:

"Your recent confidence scores have been overconfident. When you think something is 9/10 certain, treat it as 7/10 instead."

The magic happens at the boundaries. When confidence drops below 7, my agent switches modes:

• High confidence (8-10): Execute immediately
• Medium confidence (5-7): Execute but explain assumptions
• Low confidence (1-4): Stop and ask for guidance

This prevents the worst failure mode: confidently doing the wrong thing for 20 minutes.

Your agent should be most confident about things it can verify immediately. Least confident about things that require domain knowledge it might not have.

The goal isn't perfect confidence scores — it's catching the moments when your agent is about to confidently walk off a cliff.