Projection/Subscription Distribution 3.0

When Wolverine is combined with Marten into the full "Critter Stack" combination, and you're using the asynchronous projection or any event subscriptions with Marten, you can achieve potentially greater scalability for your system by better distributing the background work of these asynchronous event workers by letting Wolverine distribute the load evenly across a running cluster as shown below:

opts.Services.AddMarten(m =>
    {
        m.DisableNpgsqlLogging = true;
        m.Connection(Servers.PostgresConnectionString);
        m.DatabaseSchemaName = "csp";

        m.Projections.Add<TripProjection>(ProjectionLifecycle.Async);
        m.Projections.Add<DayProjection>(ProjectionLifecycle.Async);
        m.Projections.Add<DistanceProjection>(ProjectionLifecycle.Async);
    })
    .IntegrateWithWolverine(m =>
    {
        // This makes Wolverine distribute the registered projections
        // and event subscriptions evenly across a running application
        // cluster
        m.UseWolverineManagedEventSubscriptionDistribution = true;
    });

^{snippet source | anchor}

WARNING

This option replaces Marten's own daemon coordination — remove any AddAsyncDaemon(DaemonMode.HotCold), AddAsyncDaemon(DaemonMode.Solo), or MartenDaemonModeIsSolo() call. Wolverine sets Marten's async mode to ExternallyManaged itself, so no daemon registration is needed at all.

Combining the two would leave competing coordinators running against the same daemon, so Wolverine throws at startup rather than let the projections silently stall. See GH-3388.

With this option, Wolverine is going to ensure that every single known asynchronous event projection and every event subscription is running on exactly one running node within your application cluster. Moreover, Wolverine will purposely stop and restart projections or subscriptions to purposely spread the running load across your entire cluster of running nodes.

In the case of using multi-tenancy through separate databases per tenant with Marten, this Wolverine "agent distribution" will assign the work by tenant databases, meaning that all the running projections and subscriptions for a single tenant database will always be running on a single application node. This was done with the theory that this affinity would hopefully reduce the number of used database connections over all.

If a node is taken offline, Wolverine will detect that the node is no longer accessible and try to move start the missing projection/subscription agents on another active node.

If you run your application on only a single server, Wolverine will of course run all projections and subscriptions on just that one server.

Some other facts about this integration:

• Wolverine's agent distribution does indeed work with per-tenant database multi-tenancy
• Wolverine does automatic health checking at the running node level so that it can fail over assigned agents
• Wolverine can detect when new nodes come online and redistribute work
• Wolverine is able to support blue/green deployment and only run projections or subscriptions on active nodes where a capability is present. This just means that you can add all new projections or subscriptions, or even just new versions of a projection or subscription on some application nodes in order to do try "blue/green deployment."
• This capability does depend on Wolverine's built-in leadership election -- which fortunately got a lot better in Wolverine 3.0

Database-Affine Distribution for Multi-Database Stores 6.x

By default Wolverine spreads subscription and projection agents evenly across the cluster (with blue/green capability matching). That is the right choice for a single-database event store.

It is not the right choice for a store backed by many databases. The clearest case is a Marten store that combines sharded multi-tenancy with per-tenant event partitioning. There, many tenants are co-located in one shard database and each draws its own event sequence, so Wolverine fans agents out one-per-(shard, tenant) rather than one-per-database. With hundreds of tenants scattered across many shard databases, an even per-agent spread makes every node open a connection pool to nearly every shard database — so the pool count grows as nodes × databases and quickly exhausts a shared server's max_connections.

So Wolverine keys the distribution off the store itself, with no configuration needed: when a store reports that it is backed by multiple databases (its IEventStore.DatabaseCardinality is static- or dynamic-multiple — sharded tenancy or database-per-tenant), that store's agents are assigned with database affinity — every agent for a given database is kept together on a single node, so a node only opens pools to the databases it actually owns and the pool count scales with the number of databases, not nodes × databases. The grouping key is the [event store type]/[event store name]/[database] prefix of the agent Uri (see below), so all of a database's agents share one group. Whole groups are still spread across the cluster largest-first, so total agent counts stay balanced. A single-database store in the same application keeps the default even distribution — each store is distributed in its own pass.

Uri Structure

The Uri structure for event subscriptions or projections is:

event-subscriptions://[event store type]/[event store name]/[database server].[database name]/[relative path of the shard]

For an example from the tests: event-subscriptions://marten/main/localhost.postgres/day/all where:

• "marten" means that its a Marten based event store (we are planning on at least a SQL Server backed event store some day besides Marten)
• "main" refers to this projection being in the main DocumentStore Marten store that is added from IServiceCollection.AddMarten(). Otherwise this value would be the type name of an ancillary store type in all lower case
• "localhost" is the database server
• "postgres" is the name of the database
• "day/all" refers to a projection with the ShardName of "Day:All"

Requirements

This functionality requires Wolverine to both track running nodes and to send messages between running nodes within your clustered Wolverine service. One way or another, Wolverine needs some kind of "control queue" mechanism for this internal messaging. Not to worry though, because Wolverine will utilize in a very basic "database control queue" specifically for this if you are using the AddMarten().IntegrateWithWolverine() integration or any database backed message persistence as a default if you are not using any kind of external messaging broker that supports Wolverine control queues.

At the point of this writing, the Rabbit MQ and Azure Service Bus transport options both create a "control queue" for each executing Wolverine node that Wolverine can use for this communication in a more efficient way than the database backed control queue mechanism.

Other requirements:

• WolverineOptions.Durability.Mode must be Balanced to spread the work across multiple nodes, since that is what enables leader election and the control queue. In Solo mode every projection and subscription agent still runs — just all of them on the single node, which is why Solo is a reasonable development-time setting (see below). Serverless and MediatorOnly start no agents at all.
• In Balanced mode you cannot disable external transports with StubAllExternalTransports(), because the nodes need the control queue to communicate

If you are seeing any issues with timeouts due to the Wolverine load distribution, you can try:

1. Pre-generating any Marten types to speed up the "cold start" time
2. Use the WolverineOptions.Durability.Mode = Solo setting at development time
3. Try to use an external broker for faster communication between nodes

With Ancillary Marten Stores 5.0

Wolverine can also distribute projections and subscriptions running in ancillary stores as well. In this case, you do have to enable the Wolverine managed distribution on the main Marten store registration, but that applies to all known ancillary stores.

var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.Durability.HealthCheckPollingTime = 1.Seconds();
        opts.Durability.CheckAssignmentPeriod = 1.Seconds();

        opts.UseMessagePackSerialization();

        opts.Services.AddMarten(m =>
            {
                m.DisableNpgsqlLogging = true;
                m.Connection(Servers.PostgresConnectionString);
                m.DatabaseSchemaName = "csp2";

                m.Projections.Add<TripProjection>(ProjectionLifecycle.Async);
                m.Projections.Add<DayProjection>(ProjectionLifecycle.Async);
                m.Projections.Add<DistanceProjection>(ProjectionLifecycle.Async);
            })
            .IntegrateWithWolverine(m =>
            {
                // This makes Wolverine distribute the registered projections
                // and event subscriptions evenly across a running application
                // cluster
                m.UseWolverineManagedEventSubscriptionDistribution = true;
            });

        opts.Services.AddSingleton<ILoggerProvider>(new OutputLoggerProvider(output));

        opts.Services.AddMartenStore<ITripStore>(m =>
        {
            m.DisableNpgsqlLogging = true;
            m.Connection(Servers.PostgresConnectionString);
            m.DatabaseSchemaName = "csp3";

            m.Projections.Add<TripProjection>(ProjectionLifecycle.Async);
            m.Projections.Add<DayProjection>(ProjectionLifecycle.Async);
            m.Projections.Add<DistanceProjection>(ProjectionLifecycle.Async);
        }).IntegrateWithWolverine();

    }).StartAsync();

^{snippet source | anchor}

When a Projection Fails 6.x

A projection or subscription shard that throws while applying an event is paused by the Marten/Polecat daemon rather than skipped, unless you have opted into skipping through ErrorHandlingOptions (SkipApplyErrors and friends). A paused shard makes no further progress, and Wolverine deliberately does not restart it — restarting would fail on the exact same event, so the shard would thrash instead of advance.

Wolverine surfaces the paused shard so it does not simply go quiet:

• The agent's health check reports the failure category, the sequence number and type of the event it died on, and the root exception type — enough to act on without going to dig through logs.
• IWolverineObserver.AgentPaused(Uri agentUri, ShardFailure? failure) fires once per transition into the failed state (and again if the shard recovers and later fails anew). Implement it on a custom observer to raise your own alert; CritterWatch uses this hook.
• A NodeRecordType.AgentPaused record is written to the node-record log with the classified reason, so the failure is readable after the fact and from another process.
• IEventSubscriptionAgent.Failure exposes the same ShardFailure value directly. It is a plain, serializable record — category, the failing event, the exception message and full detail — not an Exception, so it survives being shipped to a monitoring UI.

The category tells you what to do about it:

Category	What it means
ApplyEvent	Your projection code threw on an event — the classic "poison pill". Needs a code fix, or SkipApplyErrors.
EventSerialization	The store could not deserialize or upcast a stored event body. Needs a serializer or data fix.
UnknownEventType	A stored event alias resolves to no known .NET type in this deployment — usually a missing registration or a rollback.
ProgressionOutOfOrder	The shard's progression row moved underneath it, which almost always means two processes are running the same shard.
Other	A database outage, a timeout, or a bug. No single event can be blamed.

Only Other is treated as potentially self-healing, so it is the only category Wolverine's stall detector will auto-restart. The rest are left alone until you resolve the underlying problem, at which point restarting or rewinding the agent picks it back up.

Agent Start Retries 6.x

An agent's very first assignment can race the subsystems it depends on coming up — an event-subscription shard evaluated before its store's high-water detection is running, for instance, which on a multi-store host could leave a different shard idle on every boot. Wolverine retries a failed agent start locally a couple of times before leaving it to the next assignment reevaluation:

opts.Durability.AgentStartRetryAttempts = 2;                       // default
opts.Durability.AgentStartRetryDelay = TimeSpan.FromMilliseconds(250);  // default, multiplied by attempt number

Set AgentStartRetryAttempts to 0 to disable the local retry entirely. A failure that outlives the retries is logged and picked up again on the next CheckAssignmentPeriod tick, exactly as before.