Skip to content
ai-agent.review — AI Agent Reviews, home
← All news

Research

Cisco extends FAITH to evaluate agents’ tool use and vulnerability findings

Vendor-reportedChecked: [release]

Editorial attribution: Gosia Madej

Prepared with AI assistance. Editorial attribution does not imply personal product testing or manual review of every story.

Published: · Event date:

Cisco announced Agentic FAITH on October 23. Its first integrated agentic benchmark, VulnLoc, checks both finding vulnerable files and avoiding false alarms after a patch. Execution records help distinguish a completed negative finding from a failed or unfinished run.

Vendor-reportedChecked: [release]

What changes for users

Teams evaluating tool-using agents should inspect the saved trace and completion status alongside the score. A clean command exit, an empty submission and a verified absence of a vulnerability are different outcomes. Support-agent teams would need their own task definitions and grading; this security benchmark is not a support-resolution score.

Editorial assessmentChecked: [release][quickstart]

A magnifying lens rests between two file trays, with a brass marker preserving the inspected path.
AI-assisted editorial illustration of inspecting a finding and its trace; not a Cisco interface, benchmark result or tested deployment. · AI-assisted original vector illustration for AI Agent Reviews

The extension adds multi-step, tool-using tasks to FAITH’s existing cybersecurity evaluation workflow. It records conversations, tool calls and responses, submitted answers, errors and why a task stopped. Completed-task checkpoints can support recovery when their storage remains available.

Vendor-reportedChecked: [release]

VulnLoc tests repository snapshots before and after a security fix. Phase A scores the selected file set against the expected affected files. Phase B measures correct negative findings on patched code. This is vulnerability localization, not an automatic patching service.

Vendor-reportedChecked: [release][benchmark]

The quickstart requires a read-only workspace for both phases. Its Docker smoke test runs terminal commands locally while the model runs remotely; one sampled task may still make multiple billed requests. Running query produces logs, but grading and summarizing are separate steps.

Vendor-reportedChecked: [quickstart]

Checked against sources

AI-assisted source check

How this was checked

AI-assisted checking read Cisco’s full dated announcement, the linked benchmark description, FAITH’s README and the quickstart. It traced the two phases, execution records, read-only requirement and billing warning.

Editorial assessmentChecked: [release][benchmark][quickstart][repo]

These are Cisco publications, not independent evaluations. No benchmark, container or model request was run here. No leaderboard result is reproduced, and the documentation does not establish customer-support quality or certify an agent’s security. Source screenshot capture was unavailable.

Editorial assessmentChecked: [release][benchmark][quickstart]

Source evidence

Original source excerpt. This shows what the source says; it does not establish product performance.

a successful command exit is not sufficient validation.

Read original source [quickstart]

Sources