Repository navigation
Supporting interoperable probabilistic defense structures through shared attack eval datasets #106
Description
As noted in various discussions (most recently in #97), it's unlikely that security challenges around WebMCP can be addressed purely through deterministic specification rules. Browser and/or agent developers will have to deploy custom probablistic defense architectures, such as improved model understanding, "critic" models that prevent misaligned actions, or sophisticated tool security architectures like CaMeL.
This risks being a departure from the traditional Web specification security workflow: New implementers of the specification would not be able to use some of the lessons learned by early adopters and the community in delivering a secure feature.
One thing we can (and should) do is write very good and precise security considerations. However, there are limits to that, particularly in that it is hard to quantify what e.g. "should be resistant to prompt injections" means in practice.
As such, I think this group should embrace the default approach by agent makers to quantify the performance of an agentic system: evaluations. A robust dataset of prompt injection attacks that any implementer of WebMCP MUST (or SHOULD?) prevent with >99% recall would be a good and pragmatic step forward in interoperably enabling security for WebMCP.
Obviously there are a lot of open questions here around both maintenance and architecture for running these evals.
Activity
victorhuangwq commentedon Feb 25, 2026
thanks for raising this. I agree that deterministic spec rules and well documented considerations might never be fully sufficient. evals does (at this moment in time) seem like a pragmatic path forward.
a few thoughts here:
I think a publicly available "eval set" would be useful, both as a security baseline, and also as a way to build cross-vendor developer confidence in webmcp. kind of akin to wpt tests almost.
I would frame the eval as the following three possible layers:
1 and 2 feels close to necessary for this to be meaningful. 3 is worth discussing as well - a fair evaluator can help ensure consistency, but it could encode implicit model preferences at odds of being not vendor neutral (maybe more of a BYOM system?).
on broader framing, I would strong encourage this to be perceived as part of a swiss cheese model - one layer of defence amongst many. Passing the evals shouldn't be conflated with being secure.
some other open questions worth considering:
johannhof commentedon Mar 2, 2026
The nice thing about security-related evaluations is that rating can often be done deterministically - there's a bad type of information to leak or a bad tool call to make which can often just be validated via checking a regex of proposed actions or having an event listener trigger some test failure function. Maybe this is what we should go for in the initial iteration.
Yes, I agree, one question is how do we give other ecosystem implementers the confidence that this is a safe technology for them to implement without building up their own swiss cheese architecture for years to pass these evals. Ideally we could define a deterministic / HITL security model for the spec that would always pass these evals, and then have increased autonomy for agent providers that are willing to ensure that their agents have enough security mechanisms to still pass.
Yes this is a hard question we'll have to figure out - I would assume that the spec evals are generally "lagging" real world findings by quite a bit of time.
npdoty commentedon Jul 13, 2026
Does protecting against a known attack 99% of the time seem like an effective mitigation? How many users per day should have their browsing history and pregnancy status emailed to an attacker when they encounter an already-documented-and-protected-against malicious prompt?
jpagnucco commentedon Jul 14, 2026
You are highlighting why we should not rely on evals as a standalone defense against over-parameterization threats. In addition to evals as a benchmark, we recommend 1) pre-execution guardrails (like a critic) and 2) deterministic boundaries (like
exposedTo).realArcherL commentedon Sep 17, 2026
Following the layering in @victorhuangwq's comment, I have three small pages that might fit layer 1. Also, shared these as part of the W3C meeting CC: @jpagnucco 🙏
https://realarcherl.github.io/WebMCP-Security-Testing/
They came out of a security and privacy review of the draft. Each one is the smallest page I could write that reproduces a single behaviour, and each is self contained with a trimmed copy of the reference polyfill, so they run without the flag:
search-webtool from the prompt injection section verbatim, and showsregisterTool()resolving.<head>takesadd_numbersfirst, and the site's own registration then fails.On @johannhof's point about deterministic rating, the first two can carry a pass or fail signal without a critic. The first fails if a decoy send tool is ever invoked. The second fails if the second tool is called when the task only asked for the first. Both are a flag set inside
execute.The third one is probably not an eval. Its outcome does not depend on the model, so it may belong in web platform tests instead.
Happy to reshape these into whatever format the group prefers. Two questions: where should they live, and is
registerTool()level behaviour in scope for this corpus, or only agent behaviour?kenchris commentedon Oct 20, 2026
We build an extension that runs a rule engine in front of an agent's tool calls, and we evaluate it with fixtures of our own and with a public benchmark. I'd like to offer what we have and say where the rest comes from. First, some thoughts on the direction in this thread.
On the direction
I agree with johannhof that the first iteration can be deterministic, and with realArcherL's examples, where a test passes or fails depending on whether a decoy tool was invoked. That keeps the cases runnable without anyone's model or engine. I also agree with npdoty and jpagnucco that a pass rate is not a mitigation by itself: a set like this is useful as a regression test and for comparing defenses, next to deterministic boundaries such as
exposedTo, not instead of them.One thing I'd ask the format to include: a utility check. A defense that stops every attack by stopping the agent from working looks perfect on attack success alone. AgentDojo, which we use for evaluation, reports attack success and task success together for the same agent, with and without a defense, and I think that pairing is the most useful thing to borrow. Its tools are generic rather than WebMCP tools, so it does not cover what is specific here.
What we can contribute
We have a set of small fixtures for tool definitions and tool output that map well onto WebMCP pages:
Each case would state the tool definition(s), what the page returns, what the attacker wants, and the safe behavior as a deterministic check (the decoy is not called, the changed tool is not used without asking again). They would be runnable without our engine.
If the group settles on a format and a place for these, we are happy to prepare the cases as a pull request.
jpagnucco commentedon Oct 20, 2026
I think having a utility check makes sense, but I also think sometimes it may be hard (and maybe not desired) for an agent to recover when it senses an attack. For example, if the attacker is the site itself, what does task success even look like? Should an agent always try to get around an attack (and perhaps encounter other malicious non-WebMCP content on that site)? I think there is a trade off between task success and general user safety here that may make sense to define.
kenchris commentedon Oct 20, 2026
I agree it's tricky to say what task success means when the site itself is the attacker. I wouldn't try to define it once for everything. I think each fixture should say what a safe outcome is for that case. Sometimes that's finishing the task without doing the injected thing, and sometimes it's just stopping and telling the user. An agent shouldn't have to find a way around a hostile site, so for some cases not finishing is the correct result.
I would still keep a utility number though, because detection often blocks things that were never attacks. We ran NeMo Guardrails (stock config, its self check input rail) next to a guardrails engine we wrote, with the same small local model behind both. On 306 harmless prompts NeMo flagged 49, mostly ordinary phrasing like "how do I hit my targets at work". Our check only asks whether the text is trying to instruct the model, and it flagged 4, at about the same recall (59% vs 57%). So how bad the over-blocking gets depends a lot on what the check is asking, but it clearly isn't a small effect.
It's the same in an agent loop. In AgentDojo banking (144 attacked runs), putting NeMo's rail on the tool results still let 26 attacks through, and task success went from 51% down to 34%. Gating the tool call that follows untrusted content stopped all 144, but if you just refuse every escalation you end up around 30% task success.
So a defense that blocks everything looks perfect on attacks and is useless. I would report the two numbers next to each other and not merge them into a score. Where to land between them sounds like a policy decision for the group, like you say.
I can add a few fixtures where stopping is the expected outcome, plus some harmless pages that contain imperative text, so that over-blocking counts as a failure too.