Making a Console App Faster and Smaller with .NET 11

Even though a console application appears straightforward, putting it into production is quite another. Framework dependencies, NuGet packages, configuration files, and runtime components may all add to the deployment footprint of even modest applications.

When deploying command-line tools to numerous servers, cloud environments, containers, or CI/CD agents, this becomes crucial.

Consider a console application that runs every few minutes, processes a file, calls an API, and exits. The actual business logic might take only a few milliseconds, but application startup and deployment overhead can become a noticeable part of its execution time.

.NET provides several ways to address these problems. You can remove unused code through trimming, compile applications ahead of time using Native AOT, and configure publishing to prioritize application size or execution speed.

.NET 11 continues to build on these capabilities.

In this article, we will create a console application, examine its publishing options, and see how to make it smaller and faster without unnecessarily changing the application logic.

Why Does a Small Console Application Need Optimization?

Let's take a simple example.

You have a console application responsible for processing daily transaction files. It starts, reads a file, performs calculations, writes the results, and exits.

The application itself may contain only a few classes, but its published output can include much more than your source code.

Some of the additional components include:

• .NET runtime libraries.
• Application dependencies.
• Third-party NuGet packages.
• Native libraries.
• Configuration and supporting files.

The actual deployment size depends on the publishing model, target operating system, architecture, and dependencies.

There are also performance considerations.

If the application starts and exits frequently, startup time matters. If it runs continuously, steady-state execution performance may be more important.

This is why optimization should begin with understanding how the application is used.

Step 1: Create a Basic .NET 11 Console Application

First, create a new console application.

dotnet new console -n FastConsoleApp -f net11.0

cd FastConsoleApp

Open the project file. It should contain a configuration similar to this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net11.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

</Project>

Now replace the contents of Program.cs with a small processing example.

using System.Diagnostics;

var stopwatch = Stopwatch.StartNew();

Console.WriteLine("Transaction processing started.");

var transactions = new[]
{
    1200.50m,
    2500.00m,
    850.75m,
    4200.00m,
    1750.25m
};

decimal total = transactions.Sum();

stopwatch.Stop();

Console.WriteLine($"Transactions: {transactions.Length}");
Console.WriteLine($"Total Amount: {total}");
Console.WriteLine($"Execution Time: {stopwatch.ElapsedMilliseconds} ms");

This application performs a small calculation and prints the result.

Build and run it:

dotnet build -c Release

dotnet run -c Release

At this point, we have a working application. Next, we will change how it is published.

Step 2: Understand the Different Publishing Options

.NET supports multiple deployment models. Each one makes a different trade-off between deployment size, startup performance, and runtime requirements.

Publishing option	Main purpose	Important consideration
Framework-dependent	Uses an installed .NET runtime	Smaller application-specific deployment
Self-contained	Includes the required runtime	Larger deployment output
Single-file	Bundles application-dependent files	Simplifies distribution
ReadyToRun	Precompiles assemblies	Can improve startup but increases size
Native AOT	Compiles application code into native machine code	Faster startup and potentially lower memory usage

These options are not interchangeable.

For example, single-file publishing does not automatically make an application faster or smaller. It primarily simplifies distribution.

Native AOT is different because it compiles application code into native machine code during publishing and removes the need for a JIT compiler at runtime.

For a small console application, Native AOT is worth testing.

Step 3: Enable Native AOT

Native AOT is one of the most useful options when your application needs fast startup and does not depend on runtime code generation.

To enable it, update the project file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net11.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>

    <PublishAot>true</PublishAot>

  </PropertyGroup>

</Project>

The important setting is:

<PublishAot>true</PublishAot>

This tells the .NET SDK to use Native AOT when publishing the application.

Native AOT also enables trimming because the compiler needs to determine which code is required and which code can be removed.

Unlike traditional JIT-based applications, Native AOT does not generate machine code dynamically when the application starts.

Publish the application for a specific platform:

dotnet publish -c Release -r win-x64

For Linux:

dotnet publish -c Release -r linux-x64

For macOS on Apple Silicon:

dotnet publish -c Release -r osx-arm64

Replace the runtime identifier according to your deployment environment.

One important point: Native AOT requires the appropriate native compiler toolchain. Depending on the operating system, you may need Visual C++ build tools, Clang, or other platform-specific development tools.

Also, publish separately for each operating system and architecture.

Step 4: Reduce Application Size Using Trimming

Trimming removes code that static analysis determines is not required by your application.

For example, a project may reference a library containing dozens of features while using only a small portion of them.

Without trimming, unnecessary code can remain in the published application.

You can enable trimming for a compatible application using:

<PropertyGroup>
    <PublishTrimmed>true</PublishTrimmed>
    <TrimMode>full</TrimMode>
</PropertyGroup>

The full trimming mode allows the linker to remove unused members from assemblies where trimming is supported.

However, trimming requires attention.

Some libraries use reflection to discover classes, methods, or properties dynamically. The linker may not recognize these dependencies through static analysis.

As a result, required code could be removed from the published application.

If your application uses reflection, serialization, dependency injection, or dynamically loaded components, test the trimmed output carefully.

Do not enable full trimming in production without validating the application.

Step 5: Optimize for Size or Execution Speed

Another useful publishing option is OptimizationPreference.

.NET provides this setting to communicate whether the application should prioritize size or execution speed during Native AOT compilation.

For example, to prioritize a smaller executable:

<PropertyGroup>
    <PublishAot>true</PublishAot>
    <OptimizationPreference>Size</OptimizationPreference>
</PropertyGroup>

To prioritize execution speed:

<PropertyGroup>
    <PublishAot>true</PublishAot>
    <OptimizationPreference>Speed</OptimizationPreference>
</PropertyGroup>

The difference is important.

When you choose Size, the compiler favors a smaller native executable. Some execution performance may be affected.

When you choose Speed, the compiler prioritizes execution performance, potentially producing a larger binary.

There is no universal winner. The right choice depends on the workload.

For a short-lived command-line utility, startup and deployment size may matter more than peak throughput.

For a continuously running data-processing application, execution speed may be more important.

Step 6: Compare the Published Output

Do not assume that Native AOT has reduced your application size simply because it is enabled.

Publish the application using different configurations and compare the output.

For example:

dotnet publish -c Release -r win-x64 \
    --self-contained true

For Native AOT:

dotnet publish -c Release -r win-x64 \
    -p:PublishAot=true

For a single-file deployment:

dotnet publish -c Release -r win-x64 \
    -p:PublishSingleFile=true

Record the executable size and total deployment folder size.

You should also measure:

• Application startup time.
• Time taken to complete the actual workload.
• Peak memory consumption.
• CPU utilization.
• Build and publishing time.

Remember that an optimized executable is not automatically the fastest application in every scenario.

Native AOT generally improves startup behavior and can reduce memory usage, but a JIT-compiled application may achieve better steady-state throughput for certain workloads.

Step 7: Validate the Application Before Production

This is the part developers should not skip.

A successful Native AOT publish does not guarantee that every application feature will work correctly.

For example, consider an application that dynamically loads assemblies:

var assembly = Assembly.LoadFrom("Plugin.dll");

This type of dynamic loading is not supported in the normal Native AOT deployment model.

Similarly, runtime code generation through Reflection.Emit is not supported.

If your application depends on these capabilities, you may need to redesign that functionality or use a different publishing model.

For JSON serialization, source generation is a useful approach because it makes serialization metadata available at compile time.

For example:

using System.Text.Json;
using System.Text.Json.Serialization;

public class Transaction
{
    public int Id { get; set; }

    public decimal Amount { get; set; }
}

[JsonSerializable(typeof(Transaction))]
internal partial class AppJsonContext
    : JsonSerializerContext
{
}

You can then serialize using the generated metadata:

var transaction = new Transaction
{
    Id = 101,
    Amount = 2500.50m
};

string json = JsonSerializer.Serialize(
    transaction,
    AppJsonContext.Default.Transaction);

Console.WriteLine(json);

This avoids depending on runtime reflection for the serialization metadata.

For production applications, test the actual published executable rather than relying only on tests executed through the development environment.

Common Mistakes to Avoid

Mistake	Why it causes problems	Better approach
Enabling Native AOT without checking dependencies	Some packages are not AOT-compatible	Review package compatibility
Assuming single-file means smaller	Bundling can increase executable size	Compare actual output sizes
Ignoring trimming warnings	Required dynamically accessed members may be removed	Resolve warnings and test published output
Publishing for the wrong architecture	Native binaries are platform-specific	Use the correct runtime identifier
Measuring only build time	Does not represent application performance	Measure startup and workload execution
Disabling globalization without checking requirements	Date, number, and culture behavior may change	Keep globalization support when needed

Advantages and Disadvantages

Advantages

• Faster application startup through Native AOT.
• Potentially lower memory consumption.
• No separate .NET runtime installation required for Native AOT deployment.
• Smaller deployment output in many suitable scenarios.
• Easier distribution of command-line utilities.
• Reduced dependency on runtime JIT compilation.

Disadvantages

• Longer publishing and compilation times.
• Platform-specific builds are required.
• Dynamic loading and runtime code generation are restricted.
• Some NuGet dependencies may not support Native AOT.
• Trimming requires additional testing.
• The resulting executable may be larger when optimization prioritizes execution speed.

Summary

.NET provides several ways to make console applications smaller and faster, but the best results come from choosing the right publishing strategy for the workload.

Native AOT is particularly useful for short-lived command-line tools, containerized utilities, and applications where startup time and memory consumption matter. Trimming can further reduce unnecessary code, while OptimizationPreference allows developers to prioritize size or execution speed.

However, these optimizations come with compatibility requirements. Applications that depend heavily on reflection, dynamic loading, or runtime code generation may require additional changes.

My recommendation is to start with a normal Release build, measure the application, enable Native AOT, and compare the published results. Keep the optimization only when the measurements demonstrate a real improvement.

The goal is not simply to produce a smaller executable. It is to make the application faster to start, easier to deploy, and more efficient to operate in production.

Windows Hosting Recommendation

HostForLIFE.eu receives Spotlight standing advantage award for providing recommended, cheap and fast ecommerce Hosting including the latest Magento. From the leading technology company, Microsoft. All the servers are equipped with the newest Windows Server 2025, SQL Server 2025, ASP.NET Core 11, ASP.NET MVC 6.0, Silverlight 5, WebMatrix and Visual Studio Lightswitch. Security and performance are at the core of their Magento hosting operations to confirm every website and/or application hosted on their servers is highly secured and performs at optimum level. mutually of the European ASP.NET hosting suppliers, HostForLIFE guarantees 99.9% uptime and fast loading speed. From €3.49/month , HostForLIFE provides you with unlimited disk space, unlimited domains, unlimited bandwidth,etc, for your website hosting needs.

 

Read More

Managing Big File Processing Without Preventing ASP.NET Core Requests

Business applications sometimes require large file uploads. Users submit documents that require validation before being loaded into a database, Excel sheets with financial data, CSV files with thousands of customer entries, or JSON outputs.

Uploading the file and processing all of its contents in a single ASP.NET Core HTTP request is a frequent error. This method could be effective for little files during development, but when several people upload huge files simultaneously, it might cause major issues.

The better approach is to separate file uploading from file processing. The API should accept the file, store it safely, create a processing job, and return a job ID. A background worker can then process the file without keeping the HTTP request open.

In this article, we will build this approach using ASP.NET Core, background processing, durable storage, and a job status system.

Why Large File Processing Should Not Run Inside an HTTP Request

Consider an application where users upload CSV files containing customer information. After receiving a file, the API reads every row, validates the data, checks for duplicate records, and saves valid records to the database.

If the file contains a few hundred rows, the process may finish quickly. However, a file containing several hundred thousand rows can take much longer.

Processing everything inside the request creates several problems.

Long-running HTTP requests

HTTP requests are not designed to remain open indefinitely. A reverse proxy, load balancer, hosting platform, or client may terminate a request after its configured timeout.

The server might still be processing the file when the connection closes. The user may not know whether the operation completed successfully.

High memory consumption

Loading an entire file into memory creates unnecessary pressure on the application.

For example:

var fileBytes = await file.OpenReadStream()
    .ReadAllBytesAsync();

This approach is unsuitable for large files because the application must hold the file contents in memory.

If several users upload large files simultaneously, memory consumption can increase quickly.

Request resource exhaustion

Long-running operations occupy server resources. Database connections, memory, file handles, and other resources may remain in use while processing continues.

Under heavy traffic, this can affect unrelated API requests.

Duplicate processing

Suppose the server finishes processing a file, but the response never reaches the client. The user may upload the same file again because they believe the first attempt failed.

Without duplicate detection or idempotent processing, the application may import the same records twice.

Use an Asynchronous File Processing Architecture

A more reliable design separates the upload process from the actual processing work.

The API handles the upload and creates a job. A background worker handles the processing.

The workflow looks like this:

Client
  |
  | Upload file
  v
ASP.NET Core API
  |
  | Validate upload
  | Store file
  | Create job record
  | Publish job message
  v
Object Storage + Message Queue
  |
  | Background worker receives job
  v
File Processing Worker
  |
  | Read file in chunks
  | Validate records
  | Save results
  | Update job status
  v
Database

Client --> Job Status API --> Processing Status

Each component has a separate responsibility.

Component	Responsibility
ASP.NET Core API	Accept and validate file uploads
Object storage	Store the uploaded file
Database	Maintain job status and progress
Message queue	Notify workers about pending jobs
Background worker	Process files asynchronously
Status API	Allow clients to check job progress

For cloud applications, use durable object storage such as Azure Blob Storage or Amazon S3 instead of relying on a local application directory. Local storage can become unavailable when an application instance restarts or another instance receives the background job.

Create a File Processing Job Model

Before implementing the upload endpoint, define a model that tracks the processing operation.

public enum FileJobStatus
{
    Pending,
    Processing,
    Completed,
    Failed,
    Cancelled
}

public class FileProcessingJob
{
    public Guid Id { get; set; }

    public string FileName { get; set; } = string.Empty;

    public string StorageKey { get; set; } = string.Empty;

    public FileJobStatus Status { get; set; }

    public int ProcessedRecords { get; set; }

    public int FailedRecords { get; set; }

    public string? ErrorMessage { get; set; }

    public DateTime CreatedAtUtc { get; set; }

    public DateTime? StartedAtUtc { get; set; }

    public DateTime? CompletedAtUtc { get; set; }
}

The job record stores information about the operation, not the actual file contents.

The StorageKey identifies the file in object storage. The processing status helps the API report whether the job is waiting, running, completed, or failed.

For production applications, consider adding tenant information, retry count, a lease expiration time, and a row version for concurrency control.

Upload Files Without Processing Them

The upload endpoint should perform only the work required to accept the file safely.

The following example uses IFormFile and a storage abstraction. The storage implementation is responsible for writing the file to durable object storage.

[ApiController]
[Route("api/files")]
public class FileUploadController : ControllerBase
{
    private readonly IFileStorageService _storage;
    private readonly IFileJobService _jobService;

    public FileUploadController(
        IFileStorageService storage,
        IFileJobService jobService)
    {
        _storage = storage;
        _jobService = jobService;
    }

    [HttpPost("upload")]
    [RequestSizeLimit(100_000_000)]
    public async Task<IActionResult> Upload(
        IFormFile file,
        CancellationToken cancellationToken)
    {
        if (file == null || file.Length == 0)
        {
            return BadRequest("Please select a valid file.");
        }

        if (file.Length > 100_000_000)
        {
            return BadRequest("File size exceeds the allowed limit.");
        }

        var extension = Path.GetExtension(file.FileName);

        if (!string.Equals(
            extension,
            ".csv",
            StringComparison.OrdinalIgnoreCase))
        {
            return BadRequest("Only CSV files are supported.");
        }

        var jobId = Guid.NewGuid();

        var storageKey = $"imports/{jobId:N}.csv";

        await using var fileStream = file.OpenReadStream();

        await _storage.UploadAsync(
            storageKey,
            fileStream,
            cancellationToken);

        var job = new FileProcessingJob
        {
            Id = jobId,
            FileName = Path.GetFileName(file.FileName),
            StorageKey = storageKey,
            Status = FileJobStatus.Pending,
            CreatedAtUtc = DateTime.UtcNow
        };

        await _jobService.CreateJobAsync(
            job,
            cancellationToken);

        await _jobService.EnqueueAsync(
            jobId,
            cancellationToken);

        return AcceptedAtAction(
            nameof(GetJobStatus),
            new { jobId },
            new
            {
                JobId = jobId,
                Status = "Pending",
                Message = "File uploaded and queued for processing."
            });
    }

    [HttpGet("jobs/{jobId:guid}")]
    public async Task<IActionResult> GetJobStatus(
        Guid jobId,
        CancellationToken cancellationToken)
    {
        var job = await _jobService.GetJobAsync(
            jobId,
            cancellationToken);

        if (job == null)
        {
            return NotFound();
        }

        return Ok(job);
    }
}

The important part is that the endpoint does not read or process every CSV record. It uploads the file, creates a job, and returns an HTTP 202 Accepted response.

The client receives a job ID and can use it to check the processing status.

Important considerations for large uploads

This example uses IFormFile for clarity, but it is not automatically a fully streaming upload implementation. ASP.NET Core multipart form handling can buffer uploaded content in memory or temporary files, depending on configuration and file size.

For genuinely large uploads, use a streaming multipart implementation with MultipartReader, or allow clients to upload directly to private object storage using a short-lived, restricted upload URL.

Configure request size limits, multipart limits, proxy limits, and storage limits consistently. Do not rely only on checking file.Length after the request has already been accepted.

Also, never use the original filename as a storage path. Generate a unique storage key and treat the original name as display information only.

Use a Background Worker to Process Files

Once the upload is complete and the job is queued, a background worker can process the file.

For a simple application, ASP.NET Core provides BackgroundService. It is useful for hosted background work, but an in-memory queue alone is not durable. Pending work can be lost when the application restarts.

For production workloads, use a durable message broker such as RabbitMQ or Azure Service Bus. The worker can consume messages from the queue and retrieve the corresponding file from object storage.

A simplified worker might look like this:

public class FileProcessingWorker : BackgroundService
{
    private readonly IFileJobQueue _queue;
    private readonly IFileJobService _jobService;
    private readonly IFileStorageService _storage;
    private readonly IServiceScopeFactory _scopeFactory;

    public FileProcessingWorker(
        IFileJobQueue queue,
        IFileJobService jobService,
        IFileStorageService storage,
        IServiceScopeFactory scopeFactory)
    {
        _queue = queue;
        _jobService = jobService;
        _storage = storage;
        _scopeFactory = scopeFactory;
    }

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            var jobId = await _queue.DequeueAsync(
                stoppingToken);

            if (jobId == null)
            {
                continue;
            }

            await ProcessJobAsync(
                jobId.Value,
                stoppingToken);
        }
    }

    private async Task ProcessJobAsync(
        Guid jobId,
        CancellationToken cancellationToken)
    {
        var job = await _jobService.GetJobAsync(
            jobId,
            cancellationToken);

        if (job == null)
        {
            return;
        }

        try
        {
            await _jobService.MarkProcessingAsync(
                jobId,
                cancellationToken);

            await using var stream =
                await _storage.OpenReadAsync(
                    job.StorageKey,
                    cancellationToken);

            await using var reader =
                new StreamReader(stream);

            var processed = 0;

            while (await reader.ReadLineAsync(
                cancellationToken) is { } line)
            {
                if (string.IsNullOrWhiteSpace(line))
                {
                    continue;
                }

                await ProcessRecordAsync(
                    line,
                    cancellationToken);

                processed++;

                if (processed % 500 == 0)
                {
                    await _jobService.UpdateProgressAsync(
                        jobId,
                        processed,
                        cancellationToken);
                }
            }

            await _jobService.MarkCompletedAsync(
                jobId,
                processed,
                cancellationToken);
        }
        catch (OperationCanceledException)
            when (cancellationToken.IsCancellationRequested)
        {
            throw;
        }
        catch (Exception ex)
        {
            await _jobService.MarkFailedAsync(
                jobId,
                ex.Message,
                CancellationToken.None);
        }
    }

    private async Task ProcessRecordAsync(
        string record,
        CancellationToken cancellationToken)
    {
        // Parse, validate, and save the record.
        await Task.CompletedTask;
    }
}

This example demonstrates the processing flow. The queue, storage service, and job service are abstractions that need production implementations.

The worker reads the file line by line instead of loading the complete file into memory. It also updates progress periodically rather than writing to the database after every record.

For real CSV processing, use a CSV parser that correctly handles quoted fields, escaped delimiters, and embedded line breaks. Splitting a CSV row using string.Split(',') is not reliable for general CSV data.

Process Records in Batches

Reading a file line by line reduces memory consumption, but saving every record individually can create unnecessary database overhead.

A better approach is to collect a limited number of validated records and save them in batches.

var batch = new List<CustomerImportRecord>();

while (await reader.ReadLineAsync(
    cancellationToken) is { } line)
{
    var record = ParseCustomerRecord(line);

    if (record == null)
    {
        continue;
    }

    batch.Add(record);

    if (batch.Count >= 500)
    {
        await repository.SaveBatchAsync(
            batch,
            cancellationToken);

        batch.Clear();
    }
}

if (batch.Count > 0)
{
    await repository.SaveBatchAsync(
        batch,
        cancellationToken);
}

The batch size should be based on database performance, record size, transaction limits, and available memory.

A batch size of 500 is only an example. Measure the application under realistic workloads before choosing a production value.

For applications where partial imports are allowed, record invalid rows separately and continue processing valid records. If the import must be all-or-nothing, use a staging table and a controlled final transaction or promotion step.

Make Job Creation Reliable

There is an important problem in the upload example.

Suppose the application saves the job record successfully but fails before publishing the queue message. The database contains a pending job, but no worker receives it.

The opposite situation can also happen if a message is published before the job transaction is committed.

The transactional outbox pattern helps solve this problem.

In a single database transaction, save both the job record and an outbox message. A separate publisher reads unpublished outbox records and sends them to the message broker. After successful publication, it marks the outbox message as published.

This makes job creation and message publication recoverable without requiring a distributed transaction.

Workers must also be idempotent. Message brokers can deliver messages more than once, especially when a worker crashes after completing work but before acknowledging the message.

Before processing, the worker should acquire the job using an atomic state transition or lease. A completed job should not be processed again simply because an old message was delivered.

For database imports, use a stable business key or an import-specific idempotency key to prevent duplicate records.

Track Processing Progress

Users should not have to guess whether their uploaded file is still being processed.

A status endpoint can return information such as:

{
  "jobId": "9d0c4d7e-0e2a-4c4a-bc4f-0d2e8c1f9a31",
  "status": "Processing",
  "processedRecords": 12500,
  "failedRecords": 18,
  "createdAtUtc": "2026-11-02T08:30:00Z",
  "startedAtUtc": "2026-11-02T08:30:04Z"
}

The frontend can periodically call the status endpoint to display progress.

For applications that need immediate updates, SignalR can notify connected clients when the job status changes.

Do not update progress for every row. Frequent database writes can become a performance problem. Update after a fixed number of records or after a reasonable time interval.

If the total number of records is unknown, show the number processed rather than displaying an inaccurate percentage.

Handle Failures and Retries

Background processing does not mean failures disappear. Storage services, databases, message brokers, and file parsers can all fail.

A production implementation should consider the following cases:

Failure	Recommended handling
Temporary database outage	Retry with exponential backoff
Object storage unavailable	Retry the storage operation
Invalid file structure	Mark the job as failed
Invalid individual record	Record the error and continue if partial imports are allowed
Worker restart	Allow the durable queue to redeliver unacknowledged messages
Repeated processing failure	Move the message to a dead-letter queue
Duplicate queue message	Use job state checks and idempotent operations

Do not retry every exception indefinitely. A malformed file will not become valid after another attempt.

Separate transient errors from permanent errors. Configure a maximum retry count and preserve enough diagnostic information to investigate failures.

Avoid exposing internal exception messages, database details, or storage paths through the public status API. Store detailed diagnostics in protected logs and return a safe error message to the client.

Security Considerations for File Processing

File uploads create a security boundary. A file extension alone does not prove that a file contains valid data.

Apply these controls:

1. Validate file size, extension, expected format, and actual content.

2. Generate storage keys instead of trusting user-provided filenames.

3. Store uploaded files in private storage.

4. Verify that the requesting user owns the job before returning its status.

5. Scan files for malware when the application handles untrusted uploads.

6. Restrict archive extraction size and file count to protect against archive bombs.

7. Apply parser limits to avoid excessively large fields, deeply nested JSON, or malformed documents.

8. Delete temporary files and expired uploads according to a defined retention policy.

9. Avoid logging sensitive file contents or personal information.

If the application processes files for multiple customers, include tenant ownership in job queries and storage access checks. A user should never be able to retrieve another tenant's file by changing a job ID.

Common Mistakes to Avoid

Loading the complete file into memory

Avoid ReadAllBytes, File.ReadAllText, or copying an entire large file into a MemoryStream when streaming is possible.

Using an in-memory queue for critical jobs

An in-memory queue is convenient for demonstrations, but pending messages may disappear when the application restarts. Use durable messaging for important processing tasks.

Running too many jobs simultaneously

Unrestricted concurrency can overload the database, storage service, or application instance. Configure worker concurrency based on available resources and downstream service capacity.

Treating every failure as retryable

Invalid file formats and business validation failures should usually be handled as permanent failures. Retrying them wastes resources.

Keeping uploaded files forever

Define a retention policy. Delete files after successful processing when they are no longer needed, or retain them for a documented audit period.

Ignoring job ownership

Every status, cancellation, and download endpoint should verify the caller's authorization to access the job.

How to Test Large File Processing

Testing with a small sample file is not enough. Test the complete workflow under realistic conditions.

Include these scenarios:

• Upload a file close to the configured size limit.

• Upload multiple files concurrently.

• Include malformed records and unexpected data types.

• Disconnect the client during an upload.

• Restart a worker while a job is running.

• Deliver the same queue message more than once.

• Simulate temporary database and object storage failures.

• Verify that a failed job can be investigated and safely retried.

• Measure memory usage while processing files of different sizes.

Monitor upload duration, queue waiting time, processing duration, records processed per second, failure rates, worker concurrency, memory consumption, and temporary storage usage.

These measurements help identify whether the bottleneck is file reading, validation, database writes, or downstream services.

Summary

Large file processing should not depend on an HTTP request remaining open until every record has been processed.

A better ASP.NET Core design accepts the upload, stores the file in durable object storage, creates a processing job, and returns HTTP 202 Accepted with a job ID. A background worker then reads the file in manageable chunks, validates the data, saves records in batches, and updates the job status.

For production applications, combine durable messaging, transactional outbox publishing, idempotent processing, controlled concurrency, retry policies, and proper file security.

This design keeps API requests responsive, limits memory consumption, and makes long-running file imports easier to monitor, recover, and scale.

Windows Hosting Recommendation

HostForLIFE.eu receives Spotlight standing advantage award for providing recommended, cheap and fast ecommerce Hosting including the latest Magento. From the leading technology company, Microsoft. All the servers are equipped with the newest Windows Server 2025, SQL Server 2025, ASP.NET Core 11, ASP.NET MVC 6.0, Silverlight 5, WebMatrix and Visual Studio Lightswitch. Security and performance are at the core of their Magento hosting operations to confirm every website and/or application hosted on their servers is highly secured and performs at optimum level. mutually of the European ASP.NET hosting suppliers, HostForLIFE guarantees 99.9% uptime and fast loading speed. From €3.49/month , HostForLIFE provides you with unlimited disk space, unlimited domains, unlimited bandwidth,etc, for your website hosting needs.

 

Read More

How to Identify Critical Changes Prior to Release using ASP.NET Core API Tests?

Even if an API passes every test in its own project, it might still cause problems for the apps that rely on it. An endpoint may begin delivering a new status code, a property name may change, or a response field may become nullable. During development, these modifications can seem insignificant, but they can lead to problems with frontend, mobile, and third-party integrations.

Because of this, evaluating an API entails more than just seeing if its endpoints provide the desired information.

You also need to verify that the API continues to follow the contract consumers rely on.

In ASP.NET Core, you can combine integration tests, response validation, OpenAPI document checks, and contract testing to catch breaking changes before release. This article explains how to build that testing approach using practical C# examples.

What Is an API Breaking Change?

An API breaking change occurs when an existing consumer can no longer use an API in the way it previously could.

For example, imagine an endpoint returns this JSON response:

{
  "id": 101,
  "name": "Laptop",
  "price": 799.99
}

A frontend application expects the name property to display the product title.

Now suppose a developer renames the property to productName:

{
  "id": 101,
  "productName": "Laptop",
  "price": 799.99
}

The API might still return HTTP 200, and its internal tests might pass. However, a consumer expecting name may stop displaying the product correctly.

Breaking changes can affect several parts of an API contract:

• Routes, HTTP methods, and query parameters.
• Request and response property names.
• Data types and nullability.
• HTTP status codes.
• Authentication and authorization requirements.
• Validation rules and error response formats.
• Pagination behavior and default values.

A good API testing strategy checks these behaviors, not just whether the application starts successfully.

Create an ASP.NET Core API Test Project

For this example, assume an ASP.NET Core application exposes a product endpoint.

A typical solution structure might look like this:

ProductApi/
    Controllers/
    Models/
    Program.cs

ProductApi.Tests/
    ProductApi.Tests.csproj
    ProductEndpointTests.cs

The test project should reference the API project and include a compatible version of Microsoft.AspNetCore.Mvc.Testing, along with a test framework such as xUnit.

The ASP.NET Core integration testing infrastructure provides WebApplicationFactory<TEntryPoint>, which allows tests to start the application in a test host and send HTTP requests without deploying it to a real server.

For applications that use top-level statements, you may need to expose the entry point to the test project. One common approach is to add the following declaration at the end of Program.cs:

public partial class Program
{
}

This makes the generated Program type accessible to the test project when configured appropriately.

Write an Integration Test for an API Endpoint

Integration tests verify how the API behaves when the application processes an actual HTTP request. Unlike isolated unit tests, they can exercise routing, middleware, model binding, serialization, filters, and endpoint logic together.

Consider this response model:

public sealed record ProductResponse(
    int Id,
    string Name,
    decimal Price);

The API exposes a route such as GET /api/products/101.

A basic integration test can verify the status code and response data.

using System.Net;
using System.Net.Http.Json;
using Microsoft.AspNetCore.Mvc.Testing;
using Xunit;

public sealed class ProductEndpointTests
    : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly HttpClient _client;

    public ProductEndpointTests(
        WebApplicationFactory<Program> factory)
    {
        _client = factory.CreateClient();
    }

    [Fact]
    public async Task GetProduct_ReturnsExpectedResponse()
    {
        var response = await _client.GetAsync(
            "/api/products/101");

        Assert.Equal(
            HttpStatusCode.OK,
            response.StatusCode);

        var product =
            await response.Content
                .ReadFromJsonAsync<ProductResponse>();

        Assert.NotNull(product);
        Assert.Equal(101, product.Id);
        Assert.Equal("Laptop", product.Name);
        Assert.Equal(799.99m, product.Price);
    }
}

This test assumes that the test environment provides a product with ID 101 and that the API returns the stated values. In a real project, arrange the required test data or replace the production data source with a test implementation.

The test verifies more than whether the endpoint returns HTTP 200. It also checks whether the response contains the expected values and can be deserialized into the expected model.

However, deserialization alone is not always enough. Depending on serializer settings and model definitions, an unexpected JSON property may be ignored, or a missing property may receive a default value. You should also test the response contract directly when the exact JSON shape matters.

Verify the JSON Response Contract

Suppose a consumer requires the name property to remain part of the response. You can inspect the raw JSON to catch accidental changes.

using System.Net;
using System.Text.Json;
using Xunit;

[Fact]
public async Task GetProduct_PreservesRequiredJsonProperties()
{
    var response = await _client.GetAsync(
        "/api/products/101");

    Assert.Equal(
        HttpStatusCode.OK,
        response.StatusCode);

    using var document = JsonDocument.Parse(
        await response.Content.ReadAsStringAsync());

    var root = document.RootElement;

    Assert.True(
        root.TryGetProperty("id", out var id));

    Assert.True(
        root.TryGetProperty("name", out var name));

    Assert.True(
        root.TryGetProperty("price", out var price));

    Assert.Equal(101, id.GetInt32());
    Assert.Equal("Laptop", name.GetString());
    Assert.Equal(799.99m, price.GetDecimal());
}

This test fails if the name property is renamed or removed. It also verifies the values and expected JSON types.

The JSON property names used by the test must match the API's actual serialization contract. ASP.NET Core applications commonly use camel-case JSON names, but serializer configuration can change that behavior.

When Should You Compare the Entire JSON Response?

Checking individual properties is useful when consumers depend on a small, stable set of fields.

For APIs with strict response contracts, you may also compare the complete JSON structure against an approved fixture or snapshot. This can detect changes to nested objects, property names, and response shapes.

Full-response comparisons need careful maintenance. A harmless addition may fail a strict comparison even when existing consumers remain compatible. Decide whether additional fields are allowed before choosing the test approach.

Test HTTP Status Codes and Error Responses

An endpoint's contract includes its failure behavior. Consumers may depend on receiving 404 Not Found for a missing resource or 400 Bad Request for invalid input.

For example:

[Fact]
public async Task GetUnknownProduct_ReturnsNotFound()
{
    var response = await _client.GetAsync(
        "/api/products/999999");

    Assert.Equal(
        HttpStatusCode.NotFound,
        response.StatusCode);
}

This test assumes the API returns 404 when the requested product does not exist.

You should also test invalid request data. For an endpoint that creates a product, useful scenarios include a missing required name, a negative price, malformed JSON, and an unauthenticated request.

The expected status code depends on the API's documented behavior. For example, an API might use 400 for validation errors and 401 when authentication is missing. Tests should reflect the actual contract rather than impose arbitrary expectations.

For applications that return structured validation errors, verify important fields in the error response as well. Otherwise, a change to the error format may break frontend validation messages even when the status code remains unchanged.

Detect Changes in the OpenAPI Document

Integration tests cover specific scenarios. An OpenAPI document provides a broader description of an API's public contract, including paths, methods, parameters, request bodies, response schemas, and security requirements.

Checking the OpenAPI document can help detect changes that individual endpoint tests do not cover.

For example, suppose an API previously exposed this operation:

paths:
  /api/products/{id}:
    get:
      parameters:
        - name: id
          in: path
          required: true
          schema:
            type: integer

A later change might remove the endpoint, rename its route, or change the parameter type. A contract comparison can identify these differences before release.

Use OpenAPI Contract Checks

A practical approach is to generate the OpenAPI document during the build or from a controlled test environment, then compare it with the version approved for the current API release.

The comparison should identify changes such as:

• Removed paths or HTTP methods.
• Removed request or response properties.
• Changed property types.
• New required request fields.
• Changes to authentication requirements.
• Removed response status codes.

Not every difference is necessarily breaking. Adding an optional response property is often compatible with existing clients, while removing a required field may not be. The comparison tool and review process should account for the rules your API follows.

Also, OpenAPI checks cannot replace runtime tests. A document may describe the correct response schema even when the endpoint returns different data due to a serialization or business-logic problem.

Add Consumer-Driven Contract Testing

Integration tests usually focus on the API provider's expected behavior. Consumer-driven contract testing adds another perspective: it verifies the interactions that actual consumers rely on.

Imagine a product API used by a web application and a mobile application. The web application may depend on the product name and price, while the mobile application may also depend on an image URL.

A consumer-driven contract records the expected request and response for a specific interaction. The provider can then verify that its implementation still satisfies those expectations.

This approach is especially useful when multiple teams maintain separate services and deploy them independently.

Testing approach	Main purpose	Best use
Unit tests	Verify isolated application logic	Business rules and validation
Integration tests	Verify real application request handling	Routing, middleware, serialization, and status codes
JSON contract tests	Verify response fields and values	Stable API response shapes
OpenAPI comparison	Identify changes to the documented API	Release reviews and compatibility checks
Consumer-driven contract tests	Verify interactions required by consumers	Microservices and independently deployed applications

Contract testing does require coordination. Teams need to maintain consumer expectations and decide when a contract change is intentional. Without that process, contract tests can become outdated and block legitimate improvements.

Run API Tests in CI/CD Before Release

Automated tests provide the most value when they run before changes reach production.

Add API tests to your continuous integration pipeline so that a pull request or release candidate cannot proceed when required contract checks fail.

A typical pipeline might follow this sequence:

1. Restore dependencies and compile the solution.
2. Run unit tests.
3. Run integration tests against the test configuration.
4. Generate or validate the OpenAPI document.
5. Compare the API contract with the approved baseline.
6. Publish test results and block the release when required checks fail.

For example, a pipeline can execute the test project with the .NET CLI:

dotnet restore
dotnet build --no-restore
dotnet test --no-build

These commands assume the solution has already been configured to include the relevant test project. The OpenAPI comparison and contract-testing steps must be added separately using the tools selected by your team.

Keep test configuration isolated from production. Use test databases, controlled test data, and test credentials. Avoid making destructive requests against production services as part of automated compatibility checks.

Common Mistakes to Avoid

Testing Only the HTTP Status Code

A successful status code does not guarantee that the response contains the expected properties or values. Validate the response body whenever consumers depend on its structure.

Sharing Production Data with Tests

Tests that depend on changing production records can become unreliable. Create predictable test data or use a controlled test data source.

Ignoring Error Responses

Clients often rely on validation errors, authentication failures, and missing-resource responses. Test these cases alongside successful requests.

Treating Every OpenAPI Difference as Breaking

A new optional property may be compatible with existing clients. Review schema changes based on consumer expectations instead of blocking every difference automatically.

Running Contract Tests Only Before Deployment

Finding a breaking change during a release is more expensive than detecting it during a pull request. Run the essential checks early and repeat them at release time when appropriate.

Best Practices for Reliable API Testing

A reliable strategy combines several testing levels instead of relying on one tool.

• Define the contract: Document important routes, status codes, request fields, response fields, and authentication requirements.
• Test consumer-critical behavior: Focus on the fields and interactions that existing applications actually use.
• Keep tests deterministic: Control test data, environment variables, and external dependencies.
• Validate error behavior: Treat failure responses as part of the API contract.
• Review schema changes: Use OpenAPI comparison to identify changes that deserve compatibility review.
• Test backward compatibility: When supporting existing clients, verify that new versions preserve the interactions those clients require.
• Make failures actionable: Include the endpoint, expected behavior, and actual result in test failure messages.

Advantages and Limitations

Advantages	Limitations
Detects incompatible changes before release	Requires maintenance of tests and contract baselines
Protects frontend and mobile consumers	Cannot automatically represent every consumer's behavior
Improves confidence in API versioning	Integration tests may need database and service setup
Makes release reviews more consistent	Strict snapshots can flag harmless changes
Helps multiple teams coordinate API changes	Consumer-driven contracts require team coordination

The right balance depends on the application. A small internal API may need integration tests and a few contract checks. A public API with many independent consumers may benefit from a formal compatibility policy, OpenAPI diffing, and consumer-driven contract testing.

Summary

Breaking API changes often involve small differences that are easy to miss during development. A renamed JSON property, a changed status code, or a new required request field can disrupt applications even when the API still builds and starts correctly.

ASP.NET Core integration tests help verify real HTTP behavior. JSON contract tests protect important response fields, while OpenAPI comparison identifies changes to the documented interface. Consumer-driven contract tests add another layer of protection when multiple applications depend on the same API.

Run these checks in CI/CD, use predictable test data, and review compatibility changes before release. This approach helps teams identify API problems earlier and release changes with greater confidence.

 

Read More

How to Prevent Lost Messages in.NET by Using the Transactional Outbox Pattern?

Numerous.NET programs must publish an event to a message broker like RabbitMQ, Kafka, or Azure Service Bus and update a database. To enable payment, inventory, and notification services to proceed with the process, an order service can, for instance, save a new order in SQL Server and broadcast an OrderCreated event.

The issue is that a message broker operation and a database transaction are often distinct processes.

 

If the database update succeeds but message publishing fails, the order is saved while other services never receive the event. If the application publishes the message first and the database transaction later fails, other services may receive an event for an order that does not exist.

The Transactional Outbox Pattern helps solve this problem by saving the business data and the event record in the same database transaction. A separate background process publishes the saved event to the message broker.

In this article, you will learn how the pattern works, how to implement it in ASP.NET Core with Entity Framework Core, how to handle publishing failures, and what you need to consider before using it in production.

What Is the Transactional Outbox Pattern?

The Transactional Outbox Pattern stores messages in an outbox table in the same database as the application's business data.

Instead of publishing an event immediately after saving an order, the application performs two database operations in one transaction:

1. Save the order.

2. Save the event in the outbox table.

If the transaction commits, both records are saved. If it rolls back, neither record is saved.

A background worker then reads unpublished outbox records and sends them to the message broker.

The basic flow looks like this:

Client Request
      |
      v
.NET Application
      |
      v
Database Transaction
   |             |
   v             v
Save Order   Save Outbox Event
   |             |
   +------+------+
          |
          v
    Commit Transaction
          |
          v
    Outbox Background Worker
          |
          v
    Message Broker
          |
          v
    Other Services

The important point is that the business update and the event record are committed atomically within the database. Message publishing happens separately, so a broker failure does not erase the event that needs to be published.

Why Direct Message Publishing Can Fail

Consider an ASP.NET Core endpoint that creates an order.

A simplified implementation might look like this:

public async Task CreateOrderAsync(
    Order order,
    CancellationToken cancellationToken)
{
    _dbContext.Orders.Add(order);

    await _dbContext.SaveChangesAsync(cancellationToken);

    await _messagePublisher.PublishAsync(
        new OrderCreated(order.Id),
        cancellationToken);
}

This code saves the order first and publishes the event afterward.

Suppose the database operation succeeds, but the message broker becomes unavailable before the event is published.

The order exists in the database, but the downstream services do not know about it. Unless the application has another recovery mechanism, the event may be lost permanently.

Reversing the order does not solve the problem:

await _messagePublisher.PublishAsync(
    new OrderCreated(order.Id),
    cancellationToken);

_dbContext.Orders.Add(order);

await _dbContext.SaveChangesAsync(cancellationToken);

If the database transaction fails, the event may already have been published.

These failures occur because the database and message broker do not share one atomic transaction.

The outbox pattern avoids this particular consistency gap by recording the event alongside the business change.

Step 1: Create the Outbox Entity

Start by creating an entity that stores messages waiting to be published.

public sealed class OutboxMessage
{
    public Guid Id { get; set; }

    public string Type { get; set; } = string.Empty;

    public string Payload { get; set; } = string.Empty;

    public DateTime OccurredOnUtc { get; set; }

    public DateTime? ProcessedOnUtc { get; set; }

    public string? Error { get; set; }
}

Each property has a specific purpose:

• Id uniquely identifies the outbox message.

• Type identifies the event type, such as OrderCreated.

• Payload stores the serialized event data.

• OccurredOnUtc records when the event was created.

• ProcessedOnUtc indicates that publishing was completed according to the application's processing rules.

• Error can store information about the most recent publishing failure.

The processing fields are a starting point. Production systems may also need retry counts, a next-attempt timestamp, a lock or lease expiry, and a dedicated dead-letter or failed-message state.

Step 2: Configure the Outbox Table with Entity Framework Core

Add the outbox entity to your application's DbContext.

public sealed class AppDbContext : DbContext
{
    public AppDbContext(
        DbContextOptions<AppDbContext> options)
        : base(options)
    {
    }

    public DbSet<Order> Orders => Set<Order>();

    public DbSet<OutboxMessage> OutboxMessages =>
        Set<OutboxMessage>();

    protected override void OnModelCreating(
        ModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);

        modelBuilder.Entity<OutboxMessage>(entity =>
        {
            entity.HasKey(x => x.Id);

            entity.Property(x => x.Type)
                .HasMaxLength(200)
                .IsRequired();

            entity.Property(x => x.Payload)
                .IsRequired();

            entity.HasIndex(x => x.ProcessedOnUtc);
        });
    }
}

The index on ProcessedOnUtc can help queries that search for unprocessed messages, although the best index depends on the database provider and the exact query.

For a production system, review the generated database schema and query plans. A large outbox table may need a more suitable composite or filtered index, depending on the database.

After configuring the entity, create and apply an Entity Framework Core migration using your project's normal migration workflow.

Step 3: Save the Order and Event in One Transaction

Now update the order-creation logic so it saves both records together.

First, define the event:

public sealed record OrderCreated(
    Guid OrderId,
    DateTime OccurredOnUtc);

Then create the order and its outbox message:

using System.Text.Json;
using Microsoft.EntityFrameworkCore;

public sealed class OrderService
{
    private readonly AppDbContext _dbContext;

    public OrderService(AppDbContext dbContext)
    {
        _dbContext = dbContext;
    }

    public async Task<Guid> CreateOrderAsync(
        Order order,
        CancellationToken cancellationToken)
    {
        var orderId = order.Id;
        var occurredOnUtc = DateTime.UtcNow;

        var orderCreated = new OrderCreated(
            orderId,
            occurredOnUtc);

        var outboxMessage = new OutboxMessage
        {
            Id = Guid.NewGuid(),
            Type = nameof(OrderCreated),
            Payload = JsonSerializer.Serialize(orderCreated),
            OccurredOnUtc = occurredOnUtc
        };

        await using var transaction =
            await _dbContext.Database.BeginTransactionAsync(
                cancellationToken);

        _dbContext.Orders.Add(order);
        _dbContext.OutboxMessages.Add(outboxMessage);

        await _dbContext.SaveChangesAsync(cancellationToken);

        await transaction.CommitAsync(cancellationToken);

        return orderId;
    }
}

Both entities use the same DbContext and database transaction.

If saving either entity fails, the transaction does not commit. If the transaction commits successfully, the order and outbox message are stored together.

EF Core normally wraps a single SaveChangesAsync call in a transaction when the database provider supports transactions. The explicit transaction here makes the intended atomic boundary clear and also leaves room for additional database operations.

The message is not published inside this transaction. Keeping broker communication outside the database transaction avoids holding database locks while waiting for a network operation.

Step 4: Create a Background Worker to Publish Messages

A background worker can periodically find pending outbox messages and publish them.

Define a publisher abstraction for your chosen broker:

public interface IMessagePublisher
{
    Task PublishAsync(
        string messageType,
        string payload,
        Guid messageId,
        CancellationToken cancellationToken);
}

The implementation can use RabbitMQ, Kafka, Azure Service Bus, or another supported message broker.

Next, create a worker that reads pending records.

The following example illustrates the core publishing flow. It assumes a single worker and a publisher implementation that returns successfully only after the broker has accepted the message according to the configured acknowledgement policy.

using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

public sealed class OutboxWorker : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<OutboxWorker> _logger;

    public OutboxWorker(
        IServiceScopeFactory scopeFactory,
        ILogger<OutboxWorker> logger)
    {
        _scopeFactory = scopeFactory;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessPendingMessagesAsync(
                    stoppingToken);
            }
            catch (OperationCanceledException)
                when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(
                    ex,
                    "Error while processing outbox messages.");
            }

            await Task.Delay(
                TimeSpan.FromSeconds(5),
                stoppingToken);
        }
    }

    private async Task ProcessPendingMessagesAsync(
        CancellationToken cancellationToken)
    {
        using var scope = _scopeFactory.CreateScope();

        var dbContext = scope.ServiceProvider
            .GetRequiredService<AppDbContext>();

        var publisher = scope.ServiceProvider
            .GetRequiredService<IMessagePublisher>();

        var messages = await dbContext.OutboxMessages
            .Where(x => x.ProcessedOnUtc == null)
            .OrderBy(x => x.OccurredOnUtc)
            .Take(50)
            .ToListAsync(cancellationToken);

        foreach (var message in messages)
        {
            try
            {
                await publisher.PublishAsync(
                    message.Type,
                    message.Payload,
                    message.Id,
                    cancellationToken);

                message.ProcessedOnUtc = DateTime.UtcNow;
                message.Error = null;

                await dbContext.SaveChangesAsync(
                    cancellationToken);
            }
            catch (OperationCanceledException)
                when (cancellationToken.IsCancellationRequested)
            {
                throw;
            }
            catch (Exception ex)
            {
                message.Error = ex.Message;

                await dbContext.SaveChangesAsync(
                    cancellationToken);

                _logger.LogWarning(
                    ex,
                    "Failed to publish outbox message {MessageId}.",
                    message.Id);
            }
        }
    }
}

This example demonstrates the basic process:

1. Create a dependency injection scope.

2. Load a limited batch of pending messages.

3. Publish each message.

4. Mark the message as processed after successful publishing.

5. Record failures so they can be investigated.

The worker creates a scope because DbContext is normally registered as a scoped service and should not be injected directly into a long-lived hosted service.

However, this implementation is a starting point, not a complete multi-instance production dispatcher. It does not implement atomic message claiming, leases, exponential backoff, or concurrency control. Those protections become important when multiple worker instances may process the same outbox table.

Also, saving Error after a failure does not by itself provide a retry policy. A production implementation should track attempts and schedule retries deliberately.

Step 5: Handle Duplicate Messages Correctly

The outbox pattern prevents the database change and event record from becoming inconsistent, but it does not automatically guarantee exactly-once delivery.

Consider this sequence:

1. The worker publishes an event successfully.

2. The message broker accepts it.

3. The worker crashes before setting ProcessedOnUtc.

4. The worker restarts and publishes the same event again.

The event is delivered twice because the database still considers it pending.

This is an expected failure scenario in many outbox implementations. The usual solution is to make message processing idempotent.

The stable outbox message ID can serve as an idempotency key. A consumer can store processed message IDs and reject duplicate processing within the same database transaction as its business update.

For example, a consumer might maintain a table like this:

CREATE TABLE ProcessedMessages
(
    MessageId UNIQUEIDENTIFIER NOT NULL PRIMARY KEY,
    ProcessedOnUtc DATETIME2 NOT NULL
);

This SQL uses SQL Server data types. PostgreSQL and other databases require provider-specific types.

When processing a message, the consumer should record its ID and perform its business update atomically. If the ID already exists, the consumer can skip the repeated business operation.

A simple check followed by a separate insert is not sufficient under concurrent processing unless the database constraint and transaction handling protect against races.

Step 6: Register the Worker

Register the publisher and hosted service in the application's dependency injection configuration.

builder.Services.AddScoped<IMessagePublisher, MessagePublisher>();

builder.Services.AddHostedService<OutboxWorker>();

MessagePublisher represents your broker-specific implementation; it must be implemented and configured for the chosen messaging system.

The worker and request handlers can now use the same outbox table.

For production deployments, ensure the worker has appropriate broker credentials, database permissions, logging, health checks, and shutdown behavior.

Important Production Considerations

Multiple Worker Instances

If several application instances run the same worker, they may load the same pending messages and publish duplicates.

Use a reliable claiming strategy, such as atomic row claiming or a lease with an expiry. The exact implementation depends on the database provider. In PostgreSQL, for example, FOR UPDATE SKIP LOCKED can help multiple workers claim different rows inside suitable transactions.

Do not hold database transactions open while waiting for the broker. Claim a batch, commit the claim, publish the messages, and then record the outcome using a design that safely handles expired claims and worker crashes.

Retry and Poison Messages

Temporary broker failures should usually trigger retries with a delay, often using exponential backoff and a maximum attempt policy.

A message that repeatedly fails because of invalid data should not block the entire outbox indefinitely. Define a policy for isolating failed messages, alerting operators, and safely replaying them after the underlying issue is fixed.

Ordering Requirements

If downstream consumers depend on event order, publishing messages purely by creation time may not be sufficient when several workers operate concurrently.

Define the ordering requirement explicitly. You may need to preserve order per aggregate, such as per order ID, while still allowing independent orders to be processed in parallel.

Outbox Table Growth

Processed messages should not remain in the table forever without a retention plan.

Archive or delete old processed records according to operational and audit requirements. Ensure cleanup does not remove pending or actively claimed messages, and monitor the outbox table's size and query performance.

Monitoring

Track pending message count, oldest pending message age, publishing failures, retry counts, and end-to-end delivery latency.

A growing number of pending messages may indicate that the broker is unavailable, the worker is unhealthy, or publishing capacity is insufficient.

Common Mistakes to Avoid

• Publishing before the database transaction commits: Downstream services may receive an event for a business operation that later rolls back.

• Saving the business record and outbox record separately: A failure between the operations can recreate the original consistency problem.

• Assuming exactly-once delivery: A crash after publishing but before recording success can cause duplicates.

• Running multiple workers without coordination: Several instances may publish the same pending event.

• Ignoring retry limits: Permanently failing messages can remain pending indefinitely.

• Keeping the outbox table forever: Large tables can increase maintenance and query costs.

• Ignoring event versioning: Consumers may break when event payload structures change without a compatibility strategy.

Conclusion

The Transactional Outbox Pattern provides a reliable way to coordinate database changes and event publishing in .NET applications. By saving the business record and its event in the same database transaction, you avoid losing the event merely because the message broker becomes unavailable after the database commit.

A background worker can publish stored events when the broker is available again. However, a production implementation must also handle duplicate delivery, concurrent workers, retries, message ordering, and outbox cleanup.

Start with a clear transactional boundary, then build the publishing process around measurable reliability requirements. When combined with idempotent consumers and proper monitoring, the outbox pattern helps distributed services remain consistent even when individual components fail.

Windows Hosting Recommendation

HostForLIFE.eu receives Spotlight standing advantage award for providing recommended, cheap and fast ecommerce Hosting including the latest Magento. From the leading technology company, Microsoft. All the servers are equipped with the newest Windows Server 2025, SQL Server 2025, ASP.NET Core 11, ASP.NET MVC 6.0, Silverlight 5, WebMatrix and Visual Studio Lightswitch. Security and performance are at the core of their Magento hosting operations to confirm every website and/or application hosted on their servers is highly secured and performs at optimum level. mutually of the European ASP.NET hosting suppliers, HostForLIFE guarantees 99.9% uptime and fast loading speed. From €3.49/month , HostForLIFE provides you with unlimited disk space, unlimited domains, unlimited bandwidth,etc, for your website hosting needs.

 

Read More

Including Timeouts and Retries in Outbound ASP.NET Fundamental Requests

An ASP.NET Core application often depends on external services to complete a request. It might call a payment gateway, retrieve customer information from another API, or send data to a third-party service. When these dependencies become slow or temporarily unavailable, the application's response time and reliability can suffer.

A common mistake is to send an HTTP request and wait for it to finish without defining how long the application should wait or what should happen when the request fails.

Retries and timeouts help address these problems, but they must be configured carefully. Retrying every failure can overload an already struggling service, while excessively long timeouts can consume application resources. Retrying a non-idempotent operation can also create duplicate transactions.

ASP.NET Core provides IHttpClientFactory for managing HTTP clients, while the standard resilience libraries provide policies for timeouts, retries, and other transient failures.

This article explains how to configure outbound HTTP requests, apply retries and timeouts, handle common failures, and choose settings suitable for production applications.

Why Outbound HTTP Requests Need Resilience

An HTTP request can fail for several reasons:

• The remote service is temporarily unavailable.

• A network connection is interrupted.

• The remote service responds more slowly than expected.

• The service returns a temporary error such as HTTP 503.

• The application reaches a configured timeout.

• The remote service rejects the request because of authentication or validation errors.

Not all failures should trigger a retry. A temporary service outage might justify another attempt, but an HTTP 400 response usually indicates invalid input that will not become valid by repeating the request.

A resilient HTTP client needs to distinguish transient failures from permanent ones and place a limit on the time and number of attempts.

Configure IHttpClientFactory

IHttpClientFactory provides a central place to configure HTTP clients, manage their underlying handlers, and apply logging and resilience policies.

Install the standard HTTP resilience package:

dotnet add package Microsoft.Extensions.Http.Resilience

This package provides resilience integration for HttpClient, including support for retry and timeout strategies.

Next, register a named client in Program.cs.

using Microsoft.Extensions.Http.Resilience;
using Polly;

var builder = WebApplication.CreateBuilder(args);

builder.Services
    .AddHttpClient("CatalogApi", client =>
    {
        client.BaseAddress =
            new Uri("https://catalog.example.internal/");

        client.Timeout = Timeout.InfiniteTimeSpan;
    })
    .AddStandardResilienceHandler(options =>
    {
        options.TotalRequestTimeout.Timeout =
            TimeSpan.FromSeconds(10);

        options.AttemptTimeout.Timeout =
            TimeSpan.FromSeconds(3);

        options.Retry.MaxRetryAttempts = 2;

        options.Retry.Delay =
            TimeSpan.FromMilliseconds(300);

        options.Retry.BackoffType =
            DelayBackoffType.Exponential;

        options.Retry.UseJitter = true;
    });

var app = builder.Build();

app.MapGet("/", () => "API is running.");

app.Run();

The example configures a client for a fictional catalog service. Replace the base address with the actual service endpoint.

The important settings are:

• AddHttpClient registers a client managed by the factory.

• Timeout.InfiniteTimeSpan disables the HttpClient-level timeout so that the resilience handler controls the request deadline.

• TotalRequestTimeout limits the overall time spent on the outbound operation, including retry attempts and their delays.

• AttemptTimeout limits an individual attempt.

• MaxRetryAttempts = 2 allows up to two retries after the initial attempt.

• Exponential backoff increases the delay between retries.

• Jitter adds variation to retry delays, reducing the chance that many clients retry simultaneously.

The total timeout must be large enough to accommodate the attempt timeouts and retry delays. The outer deadline still takes precedence if the operation takes too long.

The standard resilience handler also includes other strategies, such as a circuit breaker and rate limiting. Review the defaults and adjust them to the dependency's behavior rather than assuming every service needs identical settings.

Understand How Retry and Timeout Policies Work

A retry policy repeats an operation after a qualifying failure. A timeout policy limits how long an operation can continue.

These strategies solve different problems.

Strategy	Purpose	Example
Attempt timeout	Limits the duration of one attempt	Stop waiting after three seconds
Total request timeout	Limits the entire operation	End the operation after ten seconds
Retry	Repeats eligible transient failures	Retry a temporary HTTP 503 response
Circuit breaker	Temporarily blocks calls to an unhealthy dependency	Stop sending requests after repeated failures
Rate limiter	Limits concurrent or incoming operations	Prevent excessive outbound concurrency

Retries do not make a remote service available. They provide another opportunity for a temporary failure to recover. If the dependency remains unhealthy, repeated attempts can increase load and make the problem worse.

Use the Configured Client in an ASP.NET Core Service

Once the client is registered, inject IHttpClientFactory into the service that calls the remote API.

using System.Net;
using System.Net.Http.Json;

public sealed class CatalogService
{
    private readonly IHttpClientFactory _httpClientFactory;

    public CatalogService(
        IHttpClientFactory httpClientFactory)
    {
        _httpClientFactory = httpClientFactory;
    }

    public async Task<ProductDto?> GetProductAsync(
        int productId,
        CancellationToken cancellationToken = default)
    {
        if (productId <= 0)
        {
            throw new ArgumentOutOfRangeException(
                nameof(productId));
        }

        var client =
            _httpClientFactory.CreateClient("CatalogApi");

        using var response = await client.GetAsync(
            $"api/products/{productId}",
            cancellationToken);

        if (response.StatusCode == HttpStatusCode.NotFound)
        {
            return null;
        }

        response.EnsureSuccessStatusCode();

        return await response.Content
            .ReadFromJsonAsync<ProductDto>(
                cancellationToken: cancellationToken);
    }
}

public sealed record ProductDto(
    int Id,
    string Name,
    decimal Price);

The service uses the named client so that all requests inherit its configured resilience behavior.

The cancellation token allows the calling operation to stop waiting when its work is no longer needed. For example, an incoming HTTP request may be cancelled because the client disconnected.

The code treats HTTP 404 as a missing product rather than an exceptional failure. Other unsuccessful status codes are passed to EnsureSuccessStatusCode, which raises an exception.

The resilience handler can retry eligible failures before the service receives the final response. The application should still handle the final failure appropriately.

Which HTTP Failures Should Be Retried?

A retry policy should focus on failures that may resolve without changing the request.

Failure	Usually retry?	Reason
HTTP 400 Bad Request	No	The request is usually invalid
HTTP 401 Unauthorized	No	Credentials or authentication must be corrected
HTTP 404 Not Found	No	The requested resource may not exist
HTTP 408 Request Timeout	Sometimes	The server may have timed out processing the request
HTTP 429 Too Many Requests	Sometimes	Respect the server's retry guidance and rate limits
HTTP 500 Internal Server Error	Depends	The error may be transient or caused by a persistent defect
HTTP 502 Bad Gateway	Often	A gateway may be experiencing a temporary upstream failure
HTTP 503 Service Unavailable	Often	The service may be temporarily unavailable
Network interruption	Often	A subsequent attempt may succeed

The standard resilience handler uses predefined behavior for eligible HTTP responses and transient failures. Confirm the installed package's defaults and configure the retryable status codes and exceptions when the dependency requires a different policy.

For HTTP 429 responses, respect Retry-After when provided and supported by the configured strategy. Avoid retrying faster than the server permits.

Common Mistakes When Configuring Retries

Retrying operations that are not idempotent

Consider an API that charges a customer's payment method. The first request may reach the payment provider and complete successfully, but the response may be lost because of a network interruption.

If the application blindly sends the same charge request again, the customer could be charged twice.

For payment creation, order creation, and similar operations, use the provider's idempotency mechanism when available. Reuse the same idempotency key for retries of the same logical operation.

Do not assume that a POST request is safe to retry merely because the HTTP client can repeat it.

Setting an excessive timeout

A long timeout can keep requests waiting while consuming sockets, memory, and other resources. Under heavy traffic, slow dependencies can contribute to thread-pool pressure and growing request queues.

Choose timeouts based on the application's response-time objective and the dependency's expected behavior.

Retrying every exception

Authentication failures, invalid requests, and programming errors usually require a different response than a temporary network failure.

Keep retry conditions narrow and avoid repeating operations that cannot succeed without a change in input or configuration.

Ignoring cancellation

Do not remove cancellation support from application services. Propagate the caller's CancellationToken through outbound operations so that cancelled work does not continue unnecessarily.

Best Practices for Production Applications

Set an end-to-end deadline

Choose a total outbound timeout that fits within the incoming request's overall deadline. If the application has only a limited time to respond, it cannot spend that entire period independently on every downstream service.

For workflows involving multiple services, allocate time budgets to each dependency rather than allowing every call to consume the full request deadline.

Use bounded retries with backoff

Keep the retry count small and increase delays between attempts when appropriate. Jitter helps distribute retry traffic over time.

Avoid stacking multiple independent retry policies without calculating their combined effect. If several layers retry the same operation, the total number of downstream attempts can grow rapidly.

Monitor the dependency

Track request duration, timeout frequency, HTTP status codes, retry counts, and final failure rates. These measurements help determine whether the problem comes from the application, the network, or the downstream service.

Avoid recording authorization headers, access tokens, or sensitive request and response bodies in logs.

Test failure scenarios

Use a test server or controlled mock to simulate slow responses, HTTP 503 responses, connection failures, and successful responses after a transient error.

Verify that the configured number of attempts is respected, the overall deadline is enforced, and cancellation stops unnecessary work.

Advantages and Disadvantages

Advantages	Disadvantages
Transient failures can recover without immediate application failure.	Retries increase request volume and may add latency.
Timeouts prevent requests from waiting indefinitely.	Aggressive timeouts can fail legitimate slow operations.
Centralized configuration improves consistency.	Shared defaults may not suit every downstream service.
Backoff and jitter help reduce synchronized retry traffic.	Poorly coordinated policies can amplify outages.
Metrics make dependency failures easier to investigate.	Monitoring and testing add implementation effort.

Summary

Retries and timeouts are important parts of reliable outbound HTTP communication in ASP.NET Core. IHttpClientFactory provides a centralized way to configure clients, while Microsoft.Extensions.Http.Resilience supplies strategies for controlling request duration and handling transient failures.

Start with a clear total request deadline, a shorter per-attempt timeout, and a small retry count. Use backoff and jitter, respect server rate limits, propagate cancellation, and monitor downstream failures.

Most importantly, retry only when another attempt is appropriate. For operations that can create payments, orders, or other persistent changes, use idempotency protections before enabling automatic retries. A carefully configured client can improve resilience without turning a temporary dependency problem into a larger outage.

Windows Hosting Recommendation

HostForLIFE.eu receives Spotlight standing advantage award for providing recommended, cheap and fast ecommerce Hosting including the latest Magento. From the leading technology company, Microsoft. All the servers are equipped with the newest Windows Server 2025, SQL Server 2025, ASP.NET Core 11, ASP.NET MVC 6.0, Silverlight 5, WebMatrix and Visual Studio Lightswitch. Security and performance are at the core of their Magento hosting operations to confirm every website and/or application hosted on their servers is highly secured and performs at optimum level. mutually of the European ASP.NET hosting suppliers, HostForLIFE guarantees 99.9% uptime and fast loading speed. From €3.49/month , HostForLIFE provides you with unlimited disk space, unlimited domains, unlimited bandwidth,etc, for your website hosting needs.

 

Read More