Deduplication

Deduplication prevents duplicate job chains from being created. When you start a job chain with a deduplication key, Queuert checks if a chain with that key already exists and returns the existing chain instead of creating a new one.

// First call creates the chain
const chain1 = await withTransactionHooks(async (transactionHooks) =>
  client.startJobChain({
    transactionHooks,
    typeName: "sync-user",
    input: { userId: "123" },
    deduplication: { key: "sync:user:123" },
  }),
);

// Second call with same key returns existing chain
const chain2 = await withTransactionHooks(async (transactionHooks) =>
  client.startJobChain({
    transactionHooks,
    typeName: "sync-user",
    input: { userId: "123" },
    deduplication: { key: "sync:user:123" },
  }),
);

chain2.deduplicated; // true — returned existing chain
chain2.id === chain1.id; // true

Deduplication Modes

The scope option controls what jobs to check for duplicates:

• incomplete (default) — Only dedup against incomplete chains (allows new chain after previous completes)
• any — Dedup against any existing chain with this key

// Only one active health check at a time, but can start new after completion
await withTransactionHooks(async (transactionHooks) =>
  client.startJobChain({
    transactionHooks,
    typeName: "health-check",
    input: { serviceId: "api-server" },
    deduplication: {
      key: "health:api-server",
      scope: "incomplete",
    },
  }),
);

Time-Windowed Deduplication

Use windowMs to rate-limit job creation. Duplicates are prevented only within the time window.

// No duplicate syncs within 1 hour
await withTransactionHooks(async (transactionHooks) =>
  client.startJobChain({
    transactionHooks,
    typeName: "sync-data",
    input: { sourceId: "db-primary" },
    deduplication: {
      key: "sync:db-primary",
      scope: "any",
      windowMs: 60 * 60 * 1000, // 1 hour
    },
  }),
);

Excluding Chains

Use excludeJobChainIds to skip specific chains during deduplication matching. This is essential for recurring jobs that self-schedule within a completion callback — the current chain is still incomplete at that point, so without exclusion the new chain would be deduplicated against it.

// Inside a processor's completion callback
return complete(async ({ sql, transactionHooks }) => {
  await client.startJobChain({
    sql,
    transactionHooks,
    typeName: "health-check",
    input: { serviceId: job.input.serviceId },
    schedule: { afterMs: 5 * 60 * 1000 },
    deduplication: {
      key: `health:${job.input.serviceId}`,
      excludeJobChainIds: [job.chainId],
    },
  });
  return { checkedAt: new Date().toISOString() };
});

See examples/showcase-scheduling for a complete working example demonstrating deduplication with recurring jobs. See also Scheduling and Transaction Hooks.