Somewhere right now, a client is timing out on a request that succeeded. It will retry, because retrying is the correct behavior, and the server will cheerfully perform the operation again. If that operation was an email send, somebody just received the same message twice, and your logs say everything worked.
Retried requests are the normal case
Networks fail in the middle. A request can succeed on the server while the response dies on the way back: the connection drops, a proxy times out, the client process is killed mid-wait. From the client's side, these are indistinguishable from a request that never arrived, so every well-built client retries. That is not a bug in the client. A client that refuses to retry turns every network flake into a user-visible failure.
The trouble is that HTTP has no native way to say "the same request again" for anything with a side effect. A retried GET is safe by definition. A retried POST is a second POST, and the server treats it as a brand-new instruction.
What an idempotency key is
An idempotency key is a client-generated identifier, usually a UUID, sent in a header with the request. It asserts one thing: any request carrying this key is the same logical operation, no matter how many times it arrives. The client generates the key before the first attempt and reuses it on every retry of that attempt. New operation, new key. The key names the intent, not the HTTP call.
How the server should behave on replay
The first time a key arrives, the server does the work, stores the outcome against the key, and returns it. Every later arrival of the same key gets the stored response, ideally byte for byte, and triggers no work. The caller cannot tell a replayed response from the original, which is the point. "Did that send twice?" stops being a forensic question and becomes a property of the system.
The subtle case is the concurrent duplicate, where the retry lands while the original is still executing. Returning a not-found is wrong, and running the operation twice in parallel is the exact failure the key exists to prevent. Lock on the key, then either wait out the in-flight operation or return a conflict that tells the client to retry shortly. One more rule: if a key arrives with a different request body than before, reject it loudly. That is a client bug, and quietly honoring it corrupts the guarantee.
Where keys matter most
Not every route needs a key. You need one wherever a duplicate has a cost that a human will notice.
- Sends. A duplicate email is visible to someone outside your company, which makes it the most embarrassing failure available per byte.
- Bookings. Two identical bookings hold two slots on a calendar and fire two confirmations at a confused attendee.
- Payments. The textbook case. Nobody forgives a double charge, and the refund flow costs you more than the key ever will.
The details that bite later
Keys need a retention window, because storing every key forever is a slow leak. Twenty-four hours covers realistic retry behavior; whatever you choose, document it so clients know when a key expires. Scope keys per endpoint and per tenant, so one customer's collision cannot replay into another's account. And persist the key together with the operation's result in the same transaction. A crash between the two leaves you with a completed side effect and no record of it, which is the original problem wearing a new hat.
Horato accepts idempotency keys on mutating endpoints for exactly these reasons, with sends and bookings first among them. Hold your own API to the same bar: a client that retries on every timeout should be a well-behaved client, not a hazard.