Skip to content

Commit fa3c9de

Browse files
mcaxtrmcaxtr
authored
test(qa-lab): harden whatsapp qa scenarios (#95622)
* fix(whatsapp): preserve group participant identity in QA driver * fix(whatsapp): infer same-chat action targets * test(qa-lab): record whatsapp scenario posture * test(qa-lab): harden whatsapp live scenario coverage * docs(qa): describe whatsapp qa lane coverage
1 parent 4ac5cf8 commit fa3c9de

21 files changed

Lines changed: 4075 additions & 281 deletions

‎docs/channels/broadcast-groups.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ Current scope: **WhatsApp only** (web channel).
2020

2121
Broadcast groups are evaluated after channel allowlists and group activation rules. In WhatsApp groups, this means broadcasts happen when OpenClaw would normally reply (for example: on mention, depending on your group settings).
2222

23+
The live WhatsApp QA lane includes `whatsapp-broadcast-group-fanout`, which verifies that one mentioned group message can produce distinct visible replies from two configured agents.
24+
2325
## Use cases
2426

2527
<AccordionGroup>

‎docs/concepts/qa-e2e-automation.md‎

Lines changed: 52 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -735,52 +735,89 @@ Required env when `--credential-source env`:
735735
Optional:
736736

737737
- `OPENCLAW_QA_WHATSAPP_GROUP_JID` enables group scenarios such as
738-
`whatsapp-mention-gating` and `whatsapp-group-allowlist-block`.
738+
`whatsapp-mention-gating`, `whatsapp-group-pending-history-context`,
739+
`whatsapp-broadcast-group-fanout`, `whatsapp-group-activation-always`,
740+
`whatsapp-group-reply-to-bot-triggers`, group action/media/poll scenarios, and
741+
`whatsapp-group-allowlist-block`.
739742
- `OPENCLAW_QA_WHATSAPP_CAPTURE_CONTENT=1` keeps message bodies in
740743
observed-message artifacts.
741744

742745
Scenario catalog (`extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.runtime.ts`):
743746

744747
- Baseline and group gating: `whatsapp-canary`, `whatsapp-pairing-block`,
745-
`whatsapp-mention-gating`, `whatsapp-top-level-reply-shape`,
746-
`whatsapp-restart-resume`, `whatsapp-group-allowlist-block`.
748+
`whatsapp-mention-gating`, `whatsapp-group-pending-history-context`,
749+
`whatsapp-group-activation-always`,
750+
`whatsapp-group-reply-to-bot-triggers`,
751+
`whatsapp-top-level-reply-shape`, `whatsapp-restart-resume`,
752+
`whatsapp-group-allowlist-block`.
747753
- Native commands: `whatsapp-help-command`, `whatsapp-status-command`,
748754
`whatsapp-commands-command`, `whatsapp-tools-compact-command`,
749755
`whatsapp-whoami-command`, `whatsapp-context-command`,
750756
`whatsapp-native-new-command`.
751757
- Reply and final-output behavior: `whatsapp-tool-only-usage-footer`,
752758
`whatsapp-reply-to-message`, `whatsapp-group-reply-to-message`,
753-
`whatsapp-reply-context-isolation`, `whatsapp-reply-delivery-shape`,
754-
`whatsapp-stream-final-message-accounting`.
759+
`whatsapp-reply-to-mode-batched`, `whatsapp-reply-context-isolation`,
760+
`whatsapp-reply-delivery-shape`, `whatsapp-stream-final-message-accounting`.
761+
- User-path message actions: `whatsapp-agent-message-action-react` starts from
762+
a real driver DM, lets the model call the `message` tool, and observes the
763+
native WhatsApp reaction. `whatsapp-agent-message-action-upload-file` uses
764+
the same posture for `message(action=upload-file)` and observes native
765+
WhatsApp media. `whatsapp-group-agent-message-action-react` and
766+
`whatsapp-group-agent-message-action-upload-file` prove the same user-visible
767+
actions in a real WhatsApp group.
768+
- Group fanout: `whatsapp-broadcast-group-fanout` starts from one mentioned
769+
WhatsApp group message and verifies distinct visible replies from `main` and
770+
`qa-second`.
771+
- Group activation: `whatsapp-group-activation-always` changes a real group
772+
session to `/activation always`, proves an unmentioned group message wakes
773+
the agent, then restores `/activation mention`. `whatsapp-group-reply-to-bot-triggers`
774+
seeds a bot reply, sends a native quoted reply to it without an explicit
775+
mention, and verifies the agent wakes from that reply context.
755776
- Inbound media and structured messages: `whatsapp-inbound-image-caption`,
756777
`whatsapp-audio-preflight`, `whatsapp-inbound-structured-messages`,
757-
`whatsapp-group-audio-gating`. These send real WhatsApp image, audio,
758-
document, location, contact, and sticker events through the driver.
759-
- Outbound Gateway and message action coverage:
778+
`whatsapp-group-audio-gating`, `whatsapp-inbound-reaction-no-trigger`.
779+
These send real WhatsApp image, audio, document, location, contact, sticker,
780+
and reaction events through the driver.
781+
- Direct Gateway contract probes:
760782
`whatsapp-outbound-media-matrix`,
761783
`whatsapp-outbound-document-preserves-filename`, `whatsapp-outbound-poll`,
762-
`whatsapp-message-actions`.
784+
`whatsapp-group-outbound-media`, `whatsapp-group-outbound-poll`,
785+
`whatsapp-message-actions`, `whatsapp-reply-context-isolation`,
786+
`whatsapp-reply-delivery-shape`. These bypass model prompting on purpose and
787+
prove deterministic Gateway/channel `send`, `poll`, and `message.action`
788+
contracts.
763789
- Access-control coverage: `whatsapp-access-control-dm-open`,
764790
`whatsapp-access-control-dm-disabled`, `whatsapp-access-control-group-open`,
765791
`whatsapp-access-control-group-disabled`, `whatsapp-group-allowlist-block`.
766792
- Native approvals: `whatsapp-approval-exec-deny-native`,
767793
`whatsapp-approval-exec-native`, `whatsapp-approval-exec-reaction-native`,
794+
`whatsapp-approval-exec-group-reaction-native`,
768795
`whatsapp-approval-plugin-native`.
769-
- Status reactions: `whatsapp-status-reactions`.
796+
- Status reactions: `whatsapp-status-reactions`,
797+
`whatsapp-status-reaction-lifecycle`.
770798

771-
The catalog currently contains 36 scenarios. The `live-frontier` default lane is
799+
The catalog currently contains 50 scenarios. The `live-frontier` default lane is
772800
kept small at 10 scenarios for fast smoke coverage. The `mock-openai` default
773-
lane runs 31 deterministic scenarios through the real WhatsApp transport while
801+
lane runs 44 deterministic scenarios through the real WhatsApp transport while
774802
mocking only model output. Approval scenarios and a few heavier/blocking checks
775803
remain explicit by scenario id.
776804

777805
The WhatsApp QA driver observes structured live events (`text`, `media`,
778806
`location`, `reaction`, and `poll`) and can actively send media, polls,
779807
contacts, locations, and stickers. QA Lab imports that driver through the
780808
`@openclaw/whatsapp/api.js` package surface instead of reaching into private
781-
WhatsApp runtime files. Message content is redacted by default. Outbound
782-
poll and upload-file coverage run through deterministic gateway `poll` and
783-
`message.action` calls instead of model-prompt-only tool invocation.
809+
WhatsApp runtime files. For group observations, `fromJid` is the group JID while
810+
`participantJid` and `fromPhoneE164` identify the participant sender. Message
811+
content is redacted by default. Direct Gateway
812+
poll, upload-file, media, group poll, group media, and reply-shape probes are transport/API contract
813+
checks; they are not treated as proof that a user prompt made the agent choose
814+
the same action. User-path action proof comes from scenarios such as
815+
`whatsapp-agent-message-action-react` and
816+
`whatsapp-group-agent-message-action-react`, where the driver sends a normal
817+
WhatsApp message and QA Lab observes the resulting native WhatsApp artifact.
818+
WhatsApp reports include each scenario's posture (`user-path`, `direct-gateway`,
819+
or `native-approval`) so evidence cannot be mistaken for a stronger contract
820+
than it actually proves.
784821

785822
Output artifacts:
786823

‎extensions/qa-lab/src/evidence-summary.test.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,7 @@ describe("evidence summary", () => {
166166
title: "Telegram canary",
167167
status: "fail",
168168
details: "timed out waiting for SUT reply",
169+
posture: "user-path",
169170
rttMs: 4321,
170171
},
171172
],
@@ -191,6 +192,7 @@ describe("evidence summary", () => {
191192
role: "live-transport-coverage",
192193
},
193194
],
195+
posture: "user-path",
194196
execution: expect.objectContaining({
195197
runner: "crabbox",
196198
provider: {

‎extensions/qa-lab/src/evidence-summary.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,10 +180,13 @@ const qaEvidenceResultSchema = z
180180
})
181181
.strict();
182182

183+
const qaEvidencePostureSchema = z.enum(["direct-gateway", "native-approval", "user-path"]);
184+
183185
export const qaEvidenceSummaryEntrySchema = z
184186
.object({
185187
test: qaEvidenceTestSchema,
186188
coverage: z.array(qaEvidenceCoverageSchema),
189+
posture: qaEvidencePostureSchema.optional(),
187190
refs: z.array(qaEvidenceRefSchema).optional(),
188191
runtimeParityTier: nonEmptyStringSchema.optional(),
189192
execution: qaEvidenceExecutionSchema.optional(),
@@ -245,6 +248,7 @@ type QaEvidenceLiveTransportCheckInput = {
245248
title: string;
246249
status: QaEvidenceStatusInput;
247250
details: string;
251+
posture?: z.infer<typeof qaEvidencePostureSchema>;
248252
timing?: QaEvidenceTiming;
249253
rttMs?: number;
250254
rttMeasurement?: {
@@ -797,6 +801,7 @@ export function buildLiveTransportEvidenceSummary(
797801
title: check.title,
798802
},
799803
coverage,
804+
posture: check.posture,
800805
execution: {
801806
runner,
802807
environment,

‎extensions/qa-lab/src/gateway-child.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -613,6 +613,7 @@ export async function startQaGatewayChild(params: {
613613
controlUiEnabled?: boolean;
614614
enabledPluginIds?: string[];
615615
forwardHostHome?: boolean;
616+
mockAuthAgentIds?: readonly string[];
616617
mutateConfig?: (cfg: OpenClawConfig) => OpenClawConfig;
617618
runtimeEnvPatch?: NodeJS.ProcessEnv;
618619
}) {
@@ -709,6 +710,7 @@ export async function startQaGatewayChild(params: {
709710
cfg = await stageQaMockAuthProfiles({
710711
cfg,
711712
stateDir,
713+
agentIds: params.mockAuthAgentIds,
712714
providers: mockAuthProviders,
713715
});
714716
}

‎extensions/qa-lab/src/live-transports/shared/live-gateway.runtime.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ export async function startQaLiveLaneGateway(params: {
9999
thinkingDefault?: QaThinkingLevel;
100100
claudeCliAuthMode?: QaCliBackendAuthMode;
101101
controlUiEnabled?: boolean;
102+
mockAuthAgentIds?: readonly string[];
102103
mutateConfig?: (cfg: OpenClawConfig) => OpenClawConfig;
103104
}) {
104105
const mock = await startQaProviderServer(params.providerMode);
@@ -117,6 +118,7 @@ export async function startQaLiveLaneGateway(params: {
117118
thinkingDefault: params.thinkingDefault,
118119
claudeCliAuthMode: params.claudeCliAuthMode,
119120
controlUiEnabled: params.controlUiEnabled,
121+
mockAuthAgentIds: params.mockAuthAgentIds,
120122
mutateConfig: (cfg) =>
121123
prepareLiveTransportGatewayConfig(params.mutateConfig ? params.mutateConfig(cfg) : cfg),
122124
});

0 commit comments

Comments
 (0)