Where Does a Message Go When Nothing Can Handle It? Somewhere You Will Look Later

Systems frequently pass work between components as messages waiting in a queue, processed one at a time. A message that cannot be handled, because it is malformed, refers to something that no longer exists or triggers a fault, creates a dilemma. Discarding it loses work silently. Retrying it forever blocks everything behind it, which turns one bad message into a complete outage. The answer is to move it aside after a set number of failed attempts into a separate queue, so processing continues and the problem message is kept for inspection.

The contents are a catalogue of things that went wrong:

• •Messages in a format the consumer does not understand
• •References to records that were deleted in the meantime
• •Work that failed repeatedly because a dependency was down
• •Items that exceeded a size or time limit
• •Messages routed to a destination that no longer exists
• •Genuine bugs, which is why anybody looks at it at all

The mechanism fails quietly in a specific way and the failure is common enough to be a standing joke among engineers. Because the main system keeps running normally, nothing appears broken, and a queue accumulating failures for months attracts no attention until somebody asks why a customer never received something. The discipline required is therefore an alert on the queue being non-empty rather than on the queue being large, treated with the same seriousness as an outage. A holding area that nobody monitors is functionally the same as discarding the messages, with extra storage costs.

Most of the design work happens before a message ever reaches the holding area, in deciding how many attempts to make and how to space them. Retrying immediately and repeatedly hammers a struggling dependency at exactly the wrong moment, so the standard practice is to wait longer after each failure, with a random element added so that many clients do not all retry in step and produce a synchronised surge. A distinction also matters between faults that might succeed later, such as a service being briefly unavailable, and faults that never will, such as a malformed message, which should be set aside at once.

Handling the accumulated items well takes more design than the queue itself. A cause has to be diagnosed, since the same fault may have produced thousands of entries that all resolve together. Fixed messages need a route back into normal processing, which requires tooling somebody has to build. Order matters in some systems, so replaying an old message after newer ones have been processed can produce a worse state than dropping it. And the items frequently contain personal data, so a queue retained indefinitely becomes a data protection problem that nobody planned for.

Moving a repeatedly failing message aside after a set number of attempts keeps one bad item from blocking everything behind it while preserving it for inspection. The failure mode is silence, since the system looks healthy while the queue fills, so an alert on it being non-empty is essential. Replaying fixed messages needs tooling, and retained contents raise data protection questions.