Idempotency in Messaging

TIP

Wolverine's built in idempotency detection can only be used in conjunction with configured envelope storage.

Wolverine should never be trying to publish or process the exact same message at the same endpoint more than once, but it's an imperfect world and things can go weird in real life usage. Assuming that you have some sort of message storage enabled, Wolverine can use its transactional inbox support to enforce message idempotency.

As usual, we're going to reach for the EIP book and what they described as an Idempotent Receiver:

Design a receiver to be an Idempotent Receiver--one that can safely receive the same message multiple times.

In practical terms, this means that Wolverine is able to use its incoming message storage to "know" whether it has already processed an incoming message and discard any duplicate message with the same Wolverine message id that somehow, some way manages to arrive twice from an external transport. The mechanism is a little bit different depending on the Wolverine listening endpoint mode, it's always keying off the message id assigned by Wolverine.

Unless of course you're pursuing a modular monolith architecture where you might be expecting the same identified message to arrive and be processed separately in separate endpoints. In which case, this setting:

var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.PersistMessagesWithSqlServer(Servers.SqlServerConnectionString, "receiver2");
        
        // This setting changes the internal message storage identity
        opts.Durability.MessageIdentity = MessageIdentity.IdAndDestination;
    })
    .StartAsync();

^{snippet source | anchor}

Means that the uniqueness is the message id + the endpoint destination, which Wolverine stores as a Uri string in the various envelope storage databases. In all cases, Wolverine simply detects a primary key violation on the incoming envelope storage to "know" that the message has already been handled.

INFO

There are built in error policies in Wolverine (introduced in 5.3) to automatically discard any message that is determined to be a duplicate. This is done through exception filters and matching based on exceptions thrown by the underlying message storage database, and there's certainly a chance you might have to occasionally help Wolverine out with more exception filter rules to discard these messages that can never be successfully processed.

In Durable Endpoints

INFO

Wolverine 5.2 and 5.3 both included improvements to the idempotency tracking and this documentation reflects those versions. Before 5.2, Wolverine would try to mark the message as Handled after the full message was handled, but outside of any transaction during the message handling.

Idempotency checking is turned on by default with Durable endpoints. When messages are received at a Durable endpoint, this is the sequence of steps:

1. The Wolverine listener creates the Wolverine Envelope for the incoming message
2. The Wolverine listener will try to insert the new incoming Envelope into the transactional inbox storage
3. If the IMessageStore for the system throws a DuplicateIncomingEnvelopeException on that operation, that's a duplicate, so Wolverine logs that and discards that message by "ack-ing" the message broker (that's a little different based on the actual underlying message transport technology)
4. Assuming the message is correctly stored in the inbox storage, Wolverine "acks" the message with the broker and puts the message into the in memory channel for processing
5. With at least the Marten or EF Core transactional middleware support, Wolverine will try to update the storage for the current message with the status Handled as part of the message handling transaction
6. If the envelope was not previously marked as Handled, the Wolverine listener will try to mark the stored message as Handled after the message completely succeeds

Also see the later section on message retention.

Buffered or Inline Endpoints 5.3

TIP

The idempotency checking is only possible within message handlers that have the transactional middleware applied.

INFO

For Buffered or Inline endpoints, Wolverine is only storing metadata about the message and not the actual message body or Envelope. It's just enough information to feed the idempotency checks and to satisfy expected database data constraints.

Idempotency checking within message handlers executing within Buffered or more likely Inline listeners will require you to "opt in." First though, the idempotency check in this case can be done in one of two modes:

1. Eager -- just means that Wolverine will apply some middleware around the handler such that it will make an early database call to try to insert a skeleton placeholder in the transactional inbox storage
2. Optmistic -- Wolverine will try to insert the skeleton message information as part of the message handling transaction to try to avoid extra database round trips

To be honest, the EF Core integration will always use the Eager approach no matter what. Marten supports both modes, and the Optimistic approach may be valuable if all the activity of your message handler is in changes to that same database so everything can still be rolled back by the idempotency check failing.

For another example, if your message handler involves a web service call to an external system or really any kind of action that potentially makes state changes outside of the current transaction, you have to use the Eager mode.

With all of that being said, you can either opt into the idempotency checks one at a time with an overload of the [Transactional] attribute like this:

[Transactional(IdempotencyStyle.Eager)]
public static void Handle(DoSomething msg)
{
    
}

^{snippet source | anchor}

Or you can use an overload of the auto apply transactions policy:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.Policies.AutoApplyTransactions(IdempotencyStyle.Eager);
    })
    .StartAsync();

^{snippet source | anchor}

TIP

The idempotency check and the process of marking an incoming envelope are themselves "idempotent" within Wolverine to avoid Wolverine from making unnecessary database calls. ~~~~

Handled Message Retention

The way that the idempotency checks work is to keep track of messages that have already been processed in the persisted transactional inbox storage. But of course, you don't want that storage to grow forever and choke off the performance of your system, so Wolverine has a background process to delete messages marked as Handled older than a configured threshold with the setting shown below:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        // The default is 5 minutes, but if you want to keep
        // messages around longer (or shorter) in case of duplicates,
        // this is how you do it
        opts.Durability.KeepAfterMessageHandling = 10.Minutes();
    }).StartAsync();

^{snippet source | anchor}

The default is to keep messages for at least 5 minutes.