The Webhook Was Signed, Still Not Safe: A 2026 Website Security Runbook for Replay-Proof Verification

At 6:42 PM on a Friday, a payments dashboard looked normal, but support was already seeing duplicate receipts. The webhook endpoint was returning 200. Signature checks were green. Nothing looked obviously broken, until we graphed event timestamps and found a pattern: the same signed payloads were being replayed minutes later through a misconfigured retry path.

That incident changed how I think about webhook signature verification. A valid signature only tells you who signed a payload and whether bytes changed in transit. It does not prove freshness, it does not prevent duplicate processing, and it does not guarantee your framework handed you the unmodified body.

This guide is a practical 2026 runbook for hardening webhook consumers so a signed request is also a safe request.

What signatures do, and what they do not

Most providers sign webhook payloads with HMAC-based headers (for example, GitHub and Stripe docs), while broader interoperable schemes are covered by HTTP Message Signatures (RFC 9421). In all of these models, verification is only one control in a chain.

  • They do: prove payload integrity against your shared secret or public key model.
  • They do not: block replay by themselves, solve duplicate delivery semantics, or fix mutated request bodies.

So your real target is four layers together: integrity, freshness, replay resistance, and idempotent processing.

The production pattern that actually survives incidents

1) Verify against the exact raw request body

Many failures come from body parsing that changes whitespace, encoding, or key ordering before verification. Stripe explicitly warns about this, and GitHub similarly expects comparison against the raw bytes.

Use a route-level raw parser only for webhook endpoints, and perform constant-time comparison.

import express from 'express';
import crypto from 'node:crypto';

const app = express();

// Keep this route BEFORE app.use(express.json())
app.post('/webhooks/github', express.raw({ type: '*/*' }), async (req, res) => {
  const secret = process.env.GITHUB_WEBHOOK_SECRET;
  const header = req.get('x-hub-signature-256') || '';

  if (!header.startsWith('sha256=')) {
    return res.status(401).send('missing/invalid signature header');
  }

  const expected = 'sha256=' + crypto
    .createHmac('sha256', secret)
    .update(req.body) // raw Buffer
    .digest('hex');

  const ok =
    Buffer.byteLength(expected) === Buffer.byteLength(header) &&
    crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(header));

  if (!ok) return res.status(401).send('signature mismatch');

  // Freshness + replay checks come next (do not process business logic yet)
  const deliveryId = req.get('x-github-delivery');
  const event = req.get('x-github-event');

  const fresh = await isFresh(req); // provider-specific timestamp window
  const firstSeen = await claimDelivery(deliveryId, 60 * 60); // Redis SET NX EX

  if (!fresh) return res.status(401).send('stale request');
  if (!firstSeen) return res.status(202).send('duplicate delivery ignored');

  await enqueue({ deliveryId, event, body: req.body.toString('utf8') });
  return res.status(202).send('accepted');
});

app.use(express.json()); // safe for non-webhook routes

2) Enforce freshness windows

If your provider includes signed timestamps, enforce an absolute skew window (for example, 5 minutes). This sharply reduces replay usefulness. The tradeoff is clock sensitivity, so you need reliable NTP and monitoring for drift.

3) Add replay prevention storage

Store a unique delivery identifier (or a deterministic hash of signed components) in Redis with TTL. Accept only first-seen events during the replay window.

import hashlib
import redis
from datetime import datetime, timezone

r = redis.Redis(host='redis', port=6379, decode_responses=True)


def replay_key(provider: str, delivery_id: str, body: bytes) -> str:
    # Prefer provider delivery ID when available, body hash as fallback
    digest = hashlib.sha256(body).hexdigest()[:20]
    return f"wh:{provider}:{delivery_id or digest}"


def claim_once(provider: str, delivery_id: str, body: bytes, ttl_seconds: int = 3600) -> bool:
    key = replay_key(provider, delivery_id, body)
    # True only for first claim
    return bool(r.set(key, datetime.now(timezone.utc).isoformat(), nx=True, ex=ttl_seconds))

4) Keep business logic idempotent anyway

Even with replay storage, duplicates still happen in distributed failures. Your downstream write path should use idempotency keys or upsert constraints. If you need a deeper queue-first pattern, this pairs well with our earlier PHP webhook reliability post: PHP webhook idempotency, signatures, and queue-first ingestion.

Tradeoffs you should decide explicitly

  • Short replay window (more secure) vs longer tolerance (fewer false rejects): Start with 5-10 minutes if provider timestamps are stable, then tune from logs.
  • Strict reject on freshness failure vs async quarantine: Reject is cleaner; quarantine helps when upstream clocks are messy.
  • HMAC shared secret vs asymmetric signatures: HMAC is simpler operationally; asymmetric models reduce shared-secret spread but increase key lifecycle complexity.
  • Inline processing vs queue-first: Queue-first improves resilience under bursts and retries, but requires better observability and replay policy discipline.

Troubleshooting: when signatures pass but incidents still happen

Symptom 1: Same order updated multiple times

Likely cause: No replay key store, or TTL too short.
Fix: Persist provider delivery IDs for at least the provider retry horizon. For critical money flows, keep a longer dedupe ledger in your database.

Symptom 2: Signature failures only in production

Likely cause: Proxy or middleware mutating body bytes (compression, normalization, JSON parsing order).
Fix: Capture and inspect raw bytes at ingress; isolate webhook route with raw body parsing. Stripe’s docs are very explicit here.

Symptom 3: Spikes of stale timestamp rejects

Likely cause: Clock drift or delayed retries crossing freshness window.
Fix: Alert on host clock offset, verify NTP health, and temporarily widen window while root-causing delay.

Symptom 4: CPU burn during attack bursts

Likely cause: Expensive verification for obviously bad traffic.
Fix: Add cheap pre-filters first (method/path/content-type/header presence, basic rate limits), then run cryptographic checks.

How this fits into the rest of your platform security

Webhook hardening is one piece of a broader integrity posture. If you are tightening operational controls, read: security workflow drift controls. If your auth surfaces include embedded sessions, pair this with cookie boundary hygiene from our CHIPS and SameSite guide. And if your webhook actions touch repository automation, this GitHub credential migration pattern is useful too: moving from PAT scripts to GitHub App installation tokens.

FAQ

1) Is TLS enough for webhook security?

No. TLS protects transport in transit, but it does not authenticate the sender at your application boundary the way webhook signatures do. Use both.

2) Do I still need idempotency if I verify signatures and block replay?

Yes. Distributed systems can still deliver semantically duplicate events through retries, failovers, or upstream bugs. Idempotency is your final safety net.

3) Should I move immediately to HTTP Message Signatures (RFC 9421)?

Use provider-native verification first, especially when integrations already depend on it. Adopt RFC 9421 where both sides support it and you need standardized cross-provider signing semantics.

Actionable takeaways for this week

  • Put webhook endpoints on raw-body routes and verify signatures before any JSON parsing.
  • Enforce a signed timestamp freshness window and monitor clock drift.
  • Implement Redis-backed replay keys with TTL aligned to provider retry behavior.
  • Require idempotency keys or upsert constraints in downstream writes.
  • Track metrics: signature_fail_rate, replay_drop_rate, stale_timestamp_rate, and duplicate_business_event_rate.

Signed is good. Signed + fresh + first-seen + idempotent is what keeps Friday evenings quiet.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Privacy Policy · Contact · Sitemap

© 7Tech – Programming and Tech Tutorials