Webhook Reliability And Security

A junior developer does not need to design a whole event platform, but they should understand the moving parts well enough to implement a provider's webhook guide safely and review common mistakes.

Receiving and sending

Webhook work has two sides:

• Receiving webhooks means your application exposes an endpoint and another service calls it.
• Sending webhooks means your application calls a customer's or partner's endpoint when something changes.

Receiving and sending use the same HTTP concepts, but the reliability concerns are mirrored. A receiver must verify and deduplicate. A sender must sign requests, retry temporary failures, and show delivery history to support staff or customers.

Signature verification

Signatures prove that the body was produced by someone who knows the shared secret. The signature must be calculated from the raw body exactly as received. If you decode JSON and encode it again, whitespace and key order can change, which changes the signature.

<?php

declare(strict_types=1);

function signWebhook(string $timestamp, string $rawBody, string $secret): string
{
    return hash_hmac('sha256', $timestamp . '.' . $rawBody, $secret);
}

function verifyWebhookSignature(
    string $timestamp,
    string $rawBody,
    string $receivedSignature,
    string $secret
): bool {
    $expected = signWebhook($timestamp, $rawBody, $secret);

    return hash_equals($expected, $receivedSignature);
}

$body = '{"id":"evt_123","type":"invoice.paid"}';
$timestamp = '1779969600';
$secret = 'webhook_secret';
$signature = signWebhook($timestamp, $body, $secret);

var_dump(verifyWebhookSignature($timestamp, $body, $signature, $secret));

// Prints:
// bool(true)

Provider formats differ. Some send t=timestamp,v1=signature, some send plain HMAC values, and some use asymmetric signatures. Follow the provider's exact documentation.

Replay protection

A valid old webhook should not be accepted forever. If an attacker captures a signed request and sends it again months later, the signature may still match unless the scheme includes a timestamp.

<?php

declare(strict_types=1);

function timestampWithinTolerance(int $eventTimestamp, int $now, int $toleranceSeconds): bool
{
    return abs($now - $eventTimestamp) <= $toleranceSeconds;
}

$now = 1779969900;

var_dump(timestampWithinTolerance(1779969600, $now, 300));
var_dump(timestampWithinTolerance(1779960000, $now, 300));

// Prints:
// bool(true)
// bool(false)

Timestamp checks do not replace idempotency. They stop very old signed payloads being replayed; idempotency stops the same valid event being processed twice.

Idempotency and storage

The receiver should store the provider event ID with a unique constraint. The first insert accepts the event. Later inserts for the same event ID are treated as duplicates.

In plain PHP, the idea can be modelled with an array:

<?php

declare(strict_types=1);

function storeIncomingEvent(string $eventId, array &$eventStore): string
{
    if (isset($eventStore[$eventId])) {
        return 'duplicate';
    }

    $eventStore[$eventId] = [
        'status' => 'received',
        'attempts' => 0,
    ];

    return 'stored';
}

$events = [];

echo storeIncomingEvent('evt_123', $events) . PHP_EOL;
echo storeIncomingEvent('evt_123', $events) . PHP_EOL;

// Prints:
// stored
// duplicate

In a real application, this belongs in durable storage, not memory. The database record should keep the event ID, type, provider, received time, processing status, attempt count, and enough non-secret metadata to debug it.

Retry behaviour

Receivers should return a 2xx response once the event is accepted for processing. If the provider receives a timeout or 5xx, it will usually retry. If the event is malformed or the signature is invalid, retrying the same payload will not help.

Senders should retry temporary failures with backoff. They should not retry forever. After a maximum number of attempts, the delivery should be marked failed and moved to a dead-letter state where a developer or support workflow can inspect it.

<?php

declare(strict_types=1);

function deliveryState(int $statusCode, int $attempt, int $maxAttempts): string
{
    if ($statusCode >= 200 && $statusCode < 300) {
        return 'delivered';
    }

    $temporaryFailure = in_array($statusCode, [408, 429, 500, 502, 503, 504], true);

    if ($temporaryFailure && $attempt < $maxAttempts) {
        return 'retry';
    }

    return 'dead-letter';
}

echo deliveryState(503, 1, 5) . PHP_EOL;
echo deliveryState(503, 5, 5) . PHP_EOL;
echo deliveryState(400, 1, 5) . PHP_EOL;

// Prints:
// retry
// dead-letter
// dead-letter

Logs and observability

Webhook logs should help answer practical questions:

• Did the provider call the endpoint?
• Was the signature valid?
• Which event ID and type was received?
• Was it a duplicate?
• Was processing queued, completed, retried, or failed?
• Which delivery attempt failed and why?

Do not log secrets, full auth headers, payment card data, or excessive personal data. Keep the raw body only if the product, legal, and security requirements allow it.

Dead-letter handling

A dead-letter item is an event or delivery that could not be processed after the normal retry policy. It should not silently disappear.

Useful dead-letter records include:

• Event or delivery ID.
• Provider or destination.
• Event type.
• Last status code or exception class.
• Attempt count.
• First and last failure time.
• A safe summary of the failure.

A good admin or support workflow lets someone inspect the failure, fix configuration if needed, and replay the event deliberately.

Secret Rotation And Multiple Signatures

Webhook secrets sometimes need rotation after a leak, employee change, provider migration, or scheduled security policy. A receiver should be able to accept signatures made with the old and new secret during a short overlap window.

Some providers send multiple signatures or a versioned header such as t=timestamp,v1=signature. Parse the header according to the provider's documentation and compare each candidate using hash_equals().

<?php

declare(strict_types=1);

function validWithAnySecret(string $timestamp, string $body, string $signature, array $secrets): bool
{
    foreach ($secrets as $secret) {
        $expected = hash_hmac('sha256', $timestamp . '.' . $body, $secret);

        if (hash_equals($expected, $signature)) {
            return true;
        }
    }

    return false;
}

$body = '{"id":"evt_123"}';
$timestamp = '1779969600';
$signature = hash_hmac('sha256', $timestamp . '.' . $body, 'new_secret');

var_dump(validWithAnySecret($timestamp, $body, $signature, ['old_secret', 'new_secret']));

// Prints:
// bool(true)

Keep the overlap period short and observable. A permanent list of old secrets is not rotation; it is an expanding attack surface.

Atomic Idempotency Is A Database Concern

The receiver should not implement deduplication with a check-then-insert race. Two PHP workers can receive the same event at the same time, both see "not present," and both process it. A unique constraint or atomic insert prevents that.

The durable insert should happen before returning 2xx. A common table includes provider, event ID, event type, signature timestamp, received time, processing status, attempt count, and a safe payload reference or encrypted body.

The exact code depends on the database layer, but the rule is stable: the event insert and any immediate queue record should be committed together. If that commit fails, return a retryable server error so the provider tries again.

In a queue worker, business updates and marking the event processed should also be transactional where possible. If the worker crashes after the business update but before the processed mark, it may retry; the business handler must be idempotent too.

Acceptance And Processing Are Separate States

Accepted means the HTTP receiver has verified and durably stored the event. Processed means the application has completed the business work. Conflating them hides failures.

Useful states include:

• received
• queued
• processing
• processed
• ignored
• failed_retryable
• dead_letter

A duplicate received event may return 200 while the original is still processing. That is fine if the stored event remains visible and the worker can continue or retry it.

Replay Protection Has Edge Cases

Timestamp tolerance protects against very old captured payloads, but it can reject legitimate events if clocks are badly skewed. Keep server clocks synchronized with NTP and log timestamp failures with the observed skew.

Do not use a tolerance so large that captured traffic remains useful for hours. Five minutes is common, but the provider's documentation and retry behavior matter.

Replay protection and idempotency work together. A duplicate within tolerance should be accepted as duplicate. A validly signed event outside tolerance may be rejected even if the event ID is unseen, because it could be replayed traffic.

Sending Webhooks Requires Customer-Facing State

When your application sends webhooks to customers, you become the provider. Customers need a way to configure endpoints, rotate secrets, test delivery, inspect attempts, and replay failed events.

A delivery record should include destination, event ID, attempt number, response status, failure category, duration, next retry time, and a redacted response summary. Do not store customer secrets or full authorization headers in delivery logs.

Sign the raw outgoing body. Include a timestamp. Document exactly how recipients should verify it. Keep payload examples and versioning rules stable.

Retry Policy For Sending

Retry only temporary failures: timeouts, connection resets, 408, 429, 500, 502, 503, and 504 are common candidates. Do not retry 400, 401, 403, 404, or 410 forever because the destination likely needs configuration changes.

Use bounded exponential backoff with jitter and a maximum attempt count. Respect Retry-After within your maximum delay and product needs. Move exhausted deliveries to dead-letter state rather than silently dropping them.

<?php

declare(strict_types=1);

function outboundDeliveryDecision(int $statusCode, int $attempt, int $maxAttempts): string
{
    if ($statusCode >= 200 && $statusCode < 300) {
        return 'delivered';
    }

    $retryable = in_array($statusCode, [408, 429, 500, 502, 503, 504], true);

    if ($retryable && $attempt < $maxAttempts) {
        return 'retry';
    }

    return 'dead_letter';
}

foreach ([200, 503, 401] as $status) {
    echo $status . ': ' . outboundDeliveryDecision($status, 1, 3) . PHP_EOL;
}

// Prints:
// 200: delivered
// 503: retry
// 401: dead_letter

A 404 might be temporary during customer deployment, but infinite retries to a missing endpoint create noise. Let customers manually replay after they fix the destination.

Dead-Letter Is A Workflow, Not A Bin

A dead-letter record should be actionable. It should tell support or developers whether the failure was invalid configuration, repeated timeout, malformed response, internal processing error, or unsupported event data.

Provide safe replay controls. Replaying an inbound event should use the stored raw payload or normalized event record and go through the same idempotent business handler. Replaying an outbound delivery should preserve the event ID and create a new delivery attempt record, not pretend the old attempt never happened.

Access to replay tools should be permissioned and audited. A replay can send emails, grant access, issue refunds, or notify customers depending on the event type.

Logging Raw Bodies Is A Product Decision

Raw webhook bodies are useful for debugging signatures and provider disputes, but they may contain personal data, billing data, addresses, emails, or tokens. Decide retention and redaction intentionally.

A safer default is to log metadata and store raw bodies only in restricted durable storage when the integration requires it. Even then, encrypt at rest where appropriate, restrict access, and define deletion periods.

Logs should include correlation IDs, provider, event ID, event type, signature result, duplicate result, processing state, attempt count, latency, and failure category. They should not include signing secrets, full authentication headers, card data, or unnecessary personal information.

Test Failure Paths

Webhook tests should cover more than valid delivery:

• invalid signature
• old timestamp
• malformed JSON
• missing event ID
• duplicate event
• unknown event type
• durable storage failure before acknowledgement
• queue failure after insert
• worker crash after partial processing
• outbound destination timeout
• outbound permanent client error
• dead-letter replay

Use fixed clocks and recorded raw fixtures. Tests that rebuild pretty JSON from arrays cannot verify raw-body signature behavior.

What To Check

Before moving on, make sure you can:

• explain why signatures should use the raw body
• support short secret-rotation windows
• add timestamp tolerance for replay protection
• store provider event IDs with an atomic uniqueness guarantee
• separate accepted, queued, processed, ignored, failed, and dead-letter states
• describe retry policy for outgoing webhooks
• design safe delivery logs and replay controls
• avoid logging secrets or excessive personal data
• test duplicate, stale, malformed, storage-failure, and outbound-failure cases