Numerous.NET programs must publish an event to a message broker like RabbitMQ, Kafka, or Azure Service Bus and update a database. To enable payment, inventory, and notification services to proceed with the process, an order service can, for instance, save a new order in SQL Server and broadcast an OrderCreated event.
The issue is that a message broker operation and a database transaction are often distinct processes.
If the database update succeeds but message publishing fails, the order is saved while other services never receive the event. If the application publishes the message first and the database transaction later fails, other services may receive an event for an order that does not exist.
The Transactional Outbox Pattern helps solve this problem by saving the business data and the event record in the same database transaction. A separate background process publishes the saved event to the message broker.
In this article, you will learn how the pattern works, how to implement it in ASP.NET Core with Entity Framework Core, how to handle publishing failures, and what you need to consider before using it in production.
What Is the Transactional Outbox Pattern?
The Transactional Outbox Pattern stores messages in an outbox table in the same database as the application's business data.
Instead of publishing an event immediately after saving an order, the application performs two database operations in one transaction:
Save the order.
Save the event in the outbox table.
If the transaction commits, both records are saved. If it rolls back, neither record is saved.
A background worker then reads unpublished outbox records and sends them to the message broker.
The basic flow looks like this:
Client Request
|
v
.NET Application
|
v
Database Transaction
| |
v v
Save Order Save Outbox Event
| |
+------+------+
|
v
Commit Transaction
|
v
Outbox Background Worker
|
v
Message Broker
|
v
Other ServicesThe important point is that the business update and the event record are committed atomically within the database. Message publishing happens separately, so a broker failure does not erase the event that needs to be published.
Why Direct Message Publishing Can Fail
Consider an ASP.NET Core endpoint that creates an order.
A simplified implementation might look like this:
public async Task CreateOrderAsync(
Order order,
CancellationToken cancellationToken)
{
_dbContext.Orders.Add(order);
await _dbContext.SaveChangesAsync(cancellationToken);
await _messagePublisher.PublishAsync(
new OrderCreated(order.Id),
cancellationToken);
}This code saves the order first and publishes the event afterward.
Suppose the database operation succeeds, but the message broker becomes unavailable before the event is published.
The order exists in the database, but the downstream services do not know about it. Unless the application has another recovery mechanism, the event may be lost permanently.
Reversing the order does not solve the problem:
await _messagePublisher.PublishAsync(
new OrderCreated(order.Id),
cancellationToken);
_dbContext.Orders.Add(order);
await _dbContext.SaveChangesAsync(cancellationToken);If the database transaction fails, the event may already have been published.
These failures occur because the database and message broker do not share one atomic transaction.
The outbox pattern avoids this particular consistency gap by recording the event alongside the business change.
Step 1: Create the Outbox Entity
Start by creating an entity that stores messages waiting to be published.
public sealed class OutboxMessage
{
public Guid Id { get; set; }
public string Type { get; set; } = string.Empty;
public string Payload { get; set; } = string.Empty;
public DateTime OccurredOnUtc { get; set; }
public DateTime? ProcessedOnUtc { get; set; }
public string? Error { get; set; }
}Each property has a specific purpose:
Iduniquely identifies the outbox message.Typeidentifies the event type, such asOrderCreated.Payloadstores the serialized event data.OccurredOnUtcrecords when the event was created.ProcessedOnUtcindicates that publishing was completed according to the application's processing rules.Errorcan store information about the most recent publishing failure.
The processing fields are a starting point. Production systems may also need retry counts, a next-attempt timestamp, a lock or lease expiry, and a dedicated dead-letter or failed-message state.
Step 2: Configure the Outbox Table with Entity Framework Core
Add the outbox entity to your application's DbContext.
public sealed class AppDbContext : DbContext
{
public AppDbContext(
DbContextOptions<AppDbContext> options)
: base(options)
{
}
public DbSet<Order> Orders => Set<Order>();
public DbSet<OutboxMessage> OutboxMessages =>
Set<OutboxMessage>();
protected override void OnModelCreating(
ModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
modelBuilder.Entity<OutboxMessage>(entity =>
{
entity.HasKey(x => x.Id);
entity.Property(x => x.Type)
.HasMaxLength(200)
.IsRequired();
entity.Property(x => x.Payload)
.IsRequired();
entity.HasIndex(x => x.ProcessedOnUtc);
});
}
}The index on ProcessedOnUtc can help queries that search for unprocessed messages, although the best index depends on the database provider and the exact query.
For a production system, review the generated database schema and query plans. A large outbox table may need a more suitable composite or filtered index, depending on the database.
After configuring the entity, create and apply an Entity Framework Core migration using your project's normal migration workflow.
Step 3: Save the Order and Event in One Transaction
Now update the order-creation logic so it saves both records together.
First, define the event:
public sealed record OrderCreated(
Guid OrderId,
DateTime OccurredOnUtc);Then create the order and its outbox message:
using System.Text.Json;
using Microsoft.EntityFrameworkCore;
public sealed class OrderService
{
private readonly AppDbContext _dbContext;
public OrderService(AppDbContext dbContext)
{
_dbContext = dbContext;
}
public async Task<Guid> CreateOrderAsync(
Order order,
CancellationToken cancellationToken)
{
var orderId = order.Id;
var occurredOnUtc = DateTime.UtcNow;
var orderCreated = new OrderCreated(
orderId,
occurredOnUtc);
var outboxMessage = new OutboxMessage
{
Id = Guid.NewGuid(),
Type = nameof(OrderCreated),
Payload = JsonSerializer.Serialize(orderCreated),
OccurredOnUtc = occurredOnUtc
};
await using var transaction =
await _dbContext.Database.BeginTransactionAsync(
cancellationToken);
_dbContext.Orders.Add(order);
_dbContext.OutboxMessages.Add(outboxMessage);
await _dbContext.SaveChangesAsync(cancellationToken);
await transaction.CommitAsync(cancellationToken);
return orderId;
}
}Both entities use the same DbContext and database transaction.
If saving either entity fails, the transaction does not commit. If the transaction commits successfully, the order and outbox message are stored together.
EF Core normally wraps a single SaveChangesAsync call in a transaction when the database provider supports transactions. The explicit transaction here makes the intended atomic boundary clear and also leaves room for additional database operations.
The message is not published inside this transaction. Keeping broker communication outside the database transaction avoids holding database locks while waiting for a network operation.
Step 4: Create a Background Worker to Publish Messages
A background worker can periodically find pending outbox messages and publish them.
Define a publisher abstraction for your chosen broker:
public interface IMessagePublisher
{
Task PublishAsync(
string messageType,
string payload,
Guid messageId,
CancellationToken cancellationToken);
}The implementation can use RabbitMQ, Kafka, Azure Service Bus, or another supported message broker.
Next, create a worker that reads pending records.
The following example illustrates the core publishing flow. It assumes a single worker and a publisher implementation that returns successfully only after the broker has accepted the message according to the configured acknowledgement policy.
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
public sealed class OutboxWorker : BackgroundService
{
private readonly IServiceScopeFactory _scopeFactory;
private readonly ILogger<OutboxWorker> _logger;
public OutboxWorker(
IServiceScopeFactory scopeFactory,
ILogger<OutboxWorker> logger)
{
_scopeFactory = scopeFactory;
_logger = logger;
}
protected override async Task ExecuteAsync(
CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
await ProcessPendingMessagesAsync(
stoppingToken);
}
catch (OperationCanceledException)
when (stoppingToken.IsCancellationRequested)
{
break;
}
catch (Exception ex)
{
_logger.LogError(
ex,
"Error while processing outbox messages.");
}
await Task.Delay(
TimeSpan.FromSeconds(5),
stoppingToken);
}
}
private async Task ProcessPendingMessagesAsync(
CancellationToken cancellationToken)
{
using var scope = _scopeFactory.CreateScope();
var dbContext = scope.ServiceProvider
.GetRequiredService<AppDbContext>();
var publisher = scope.ServiceProvider
.GetRequiredService<IMessagePublisher>();
var messages = await dbContext.OutboxMessages
.Where(x => x.ProcessedOnUtc == null)
.OrderBy(x => x.OccurredOnUtc)
.Take(50)
.ToListAsync(cancellationToken);
foreach (var message in messages)
{
try
{
await publisher.PublishAsync(
message.Type,
message.Payload,
message.Id,
cancellationToken);
message.ProcessedOnUtc = DateTime.UtcNow;
message.Error = null;
await dbContext.SaveChangesAsync(
cancellationToken);
}
catch (OperationCanceledException)
when (cancellationToken.IsCancellationRequested)
{
throw;
}
catch (Exception ex)
{
message.Error = ex.Message;
await dbContext.SaveChangesAsync(
cancellationToken);
_logger.LogWarning(
ex,
"Failed to publish outbox message {MessageId}.",
message.Id);
}
}
}
}This example demonstrates the basic process:
Create a dependency injection scope.
Load a limited batch of pending messages.
Publish each message.
Mark the message as processed after successful publishing.
Record failures so they can be investigated.
The worker creates a scope because DbContext is normally registered as a scoped service and should not be injected directly into a long-lived hosted service.
However, this implementation is a starting point, not a complete multi-instance production dispatcher. It does not implement atomic message claiming, leases, exponential backoff, or concurrency control. Those protections become important when multiple worker instances may process the same outbox table.
Also, saving Error after a failure does not by itself provide a retry policy. A production implementation should track attempts and schedule retries deliberately.
Step 5: Handle Duplicate Messages Correctly
The outbox pattern prevents the database change and event record from becoming inconsistent, but it does not automatically guarantee exactly-once delivery.
Consider this sequence:
The worker publishes an event successfully.
The message broker accepts it.
The worker crashes before setting
ProcessedOnUtc.The worker restarts and publishes the same event again.
The event is delivered twice because the database still considers it pending.
This is an expected failure scenario in many outbox implementations. The usual solution is to make message processing idempotent.
The stable outbox message ID can serve as an idempotency key. A consumer can store processed message IDs and reject duplicate processing within the same database transaction as its business update.
For example, a consumer might maintain a table like this:
CREATE TABLE ProcessedMessages
(
MessageId UNIQUEIDENTIFIER NOT NULL PRIMARY KEY,
ProcessedOnUtc DATETIME2 NOT NULL
);This SQL uses SQL Server data types. PostgreSQL and other databases require provider-specific types.
When processing a message, the consumer should record its ID and perform its business update atomically. If the ID already exists, the consumer can skip the repeated business operation.
A simple check followed by a separate insert is not sufficient under concurrent processing unless the database constraint and transaction handling protect against races.
Step 6: Register the Worker
Register the publisher and hosted service in the application's dependency injection configuration.
builder.Services.AddScoped<IMessagePublisher, MessagePublisher>();
builder.Services.AddHostedService<OutboxWorker>();MessagePublisher represents your broker-specific implementation; it must be implemented and configured for the chosen messaging system.
The worker and request handlers can now use the same outbox table.
For production deployments, ensure the worker has appropriate broker credentials, database permissions, logging, health checks, and shutdown behavior.
Important Production Considerations
Multiple Worker Instances
If several application instances run the same worker, they may load the same pending messages and publish duplicates.
Use a reliable claiming strategy, such as atomic row claiming or a lease with an expiry. The exact implementation depends on the database provider. In PostgreSQL, for example, FOR UPDATE SKIP LOCKED can help multiple workers claim different rows inside suitable transactions.
Do not hold database transactions open while waiting for the broker. Claim a batch, commit the claim, publish the messages, and then record the outcome using a design that safely handles expired claims and worker crashes.
Retry and Poison Messages
Temporary broker failures should usually trigger retries with a delay, often using exponential backoff and a maximum attempt policy.
A message that repeatedly fails because of invalid data should not block the entire outbox indefinitely. Define a policy for isolating failed messages, alerting operators, and safely replaying them after the underlying issue is fixed.
Ordering Requirements
If downstream consumers depend on event order, publishing messages purely by creation time may not be sufficient when several workers operate concurrently.
Define the ordering requirement explicitly. You may need to preserve order per aggregate, such as per order ID, while still allowing independent orders to be processed in parallel.
Outbox Table Growth
Processed messages should not remain in the table forever without a retention plan.
Archive or delete old processed records according to operational and audit requirements. Ensure cleanup does not remove pending or actively claimed messages, and monitor the outbox table's size and query performance.
Monitoring
Track pending message count, oldest pending message age, publishing failures, retry counts, and end-to-end delivery latency.
A growing number of pending messages may indicate that the broker is unavailable, the worker is unhealthy, or publishing capacity is insufficient.
Common Mistakes to Avoid
Publishing before the database transaction commits: Downstream services may receive an event for a business operation that later rolls back.
Saving the business record and outbox record separately: A failure between the operations can recreate the original consistency problem.
Assuming exactly-once delivery: A crash after publishing but before recording success can cause duplicates.
Running multiple workers without coordination: Several instances may publish the same pending event.
Ignoring retry limits: Permanently failing messages can remain pending indefinitely.
Keeping the outbox table forever: Large tables can increase maintenance and query costs.
Ignoring event versioning: Consumers may break when event payload structures change without a compatibility strategy.
Conclusion
The Transactional Outbox Pattern provides a reliable way to coordinate database changes and event publishing in .NET applications. By saving the business record and its event in the same database transaction, you avoid losing the event merely because the message broker becomes unavailable after the database commit.
A background worker can publish stored events when the broker is available again. However, a production implementation must also handle duplicate delivery, concurrent workers, retries, message ordering, and outbox cleanup.
Start with a clear transactional boundary, then build the publishing process around measurable reliability requirements. When combined with idempotent consumers and proper monitoring, the outbox pattern helps distributed services remain consistent even when individual components fail.
Windows Hosting Recommendation
