Duplicate webhook events: idempotency and retry handling

How duplicates happen

• The receiver completes work but the response is lost.
• The handler returns a non-2xx response after partially changing state.
• Processing exceeds the sender timeout and a retry overlaps the first attempt.
• An operator manually redelivers an event.
• Two related but distinct events look identical because the handler uses the wrong deduplication key.

Choose the idempotency key

Prefer a stable provider event or delivery ID. Scope it by provider or tenant if IDs are not globally unique. Do not deduplicate only by payload hash when legitimate repeated events can have identical bodies.

Atomic processing pattern

BEGIN;

INSERT INTO webhook_inbox (provider, event_id, payload)
VALUES ('provider', 'evt_123', '{...}')
ON CONFLICT (provider, event_id) DO NOTHING;

-- Continue only when the insert created a new row.
-- Apply the business change in the same transaction when practical.

COMMIT;

Diagnosis procedure

1. Group delivery attempts by provider event ID or delivery ID.
2. Compare timestamps and response statuses to identify why a retry occurred.
3. Check whether each attempt passed signature verification.
4. Trace each attempt to database writes, queue messages, emails, or other side effects.
5. Add a unique constraint or atomic claim step before applying the side effect.
6. Replay the same event twice in a test environment and confirm the final state is unchanged after the first success.

Do not reject harmless duplicates

After verifying the request, returning 2xx for an already processed event usually prevents unnecessary retries. Record the duplicate for observability, but do not repeat the business action.