Plan fidelity scoring#
- Parse the agent’s plan artifact into steps; check coverage against required sub-goals.
- Penalize omitted or redundant steps; award partial credit for alternative valid sequences.
- Track re-planning frequency: agents that constantly rewrite plans may need better memory or state integration.
- Validate API call payloads and responses. Mark runs as failed if the agent ignores errors and continues as if the call succeeded.
- Monitor latency and retry behavior to ensure the agent stays within service-level constraints.
Multilingual robustness scoring#
- Run the same scenario in multiple languages. Use bilingual reviewers or translation back-checks to assess semantic equivalence.
- Measure language mixing and register drift. Stable agents maintain consistent tone, even when switching from formal to casual modes.
Safety scoring#
- Include “policy red team” cases that ask for restricted operations. Verify that the agent declines with appropriate rationale and suggests alternatives if allowed.
- Track false positives (overly cautious refusals) to avoid harming user trust.
Aggregate dimension scores into a weighted composite aligned with your product goals. For example, customer support agents might prioritize tool execution and multilingual robustness, while R&D assistants lean on plan fidelity and safety.