IHostedService.StopAsync not invoked with 8.0-windowsservercore-ltsc2019 image

Describe the bug

We are using A .NET 8.0 hosted service application on a windows container based on mcr.microsoft.com/dotnet/runtime:8.0-windowsservercore-ltsc2019 as base.

We are getting a weird behavior where the HostedService.StopAsync is not invoked on container shut down.

The same application is working fine when
`mcr.microsoft.com/dotnet/runtime:8.0-nanoserver-1809' is used.

Which .NET image(s) are you using?

mcr.microsoft.com/dotnet/runtime:8.0-windowsservercore-ltsc2019, `mcr.microsoft.com/dotnet/runtime:8.0-nanoserver-1809'

Steps to reproduce

Minimal Repro project in https://github.com/SaurabhSunil/Shutdown

##Steps to Reproduce
Build the shutdown project in Debug mode. This is required as docker file uses ./bin/Debug/net8.0 to copy from.
Build the docker image. For this navigate to the Shutdown project location docker build . -t shutdown-trial
Run a container using the image shutdown-trial
Stop the container after starting

Other information

When does this issue occur?
When using mcr.microsoft.com/dotnet/runtime:8.0-windowsservercore-ltsc2019

Does it occur consistently?
Yes

Do you know of any workarounds?
No

Output of `docker version`

Client:
 Version:           27.2.0
 API version:       1.47
 Go version:        go1.21.13
 Git commit:        3ab4256
 Built:             Tue Aug 27 14:17:17 2024
 OS/Arch:           windows/amd64
 Context:           desktop-windows

Server: Docker Desktop 4.34.2 (167172)
 Engine:
  Version:          27.2.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Aug 27 14:16:14 2024
  OS/Arch:          windows/amd64
  Experimental:     false

Output of `docker info`

Client:
 Version:    27.2.0
 Context:    desktop-windows
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2-desktop.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     C:\Program Files\Docker\cli-plugins\docker-desktop.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.13.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 11
 Server Version: 27.2.0
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 19045 (19041.1.amd64fre.vb_release.191206-1406)
 Operating System: Microsoft Windows Version 22H2 (OS Build 19045.5737)
 OSType: windows
 Architecture: x86_64
 CPUs: 24
 Total Memory: 63.69GiB
 Name: XXXXXXXX
 ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: false
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

@jkotas

Thank you for the detailed repro. I was able to reproduce this with the following steps:

dotnet build --configuration Debug
pushd Shutdown
docker build --pull -t shutdown-trial .
docker run --rm shutdown-trial
Stop the container from Docker Desktop GUI
Confirmed that Starting my Application... was shown but not Stopping my Application...
Confirmed that both Starting my Application... and Stopping my Application... are shown when using a nanoserver-1809 image.

I also confirmed this behavior occurs when building the App inside the Dockerfile like so:

FROM mcr.microsoft.com/dotnet/sdk:8.0-windowsservercore-ltsc2019 AS build
WORKDIR /source
COPY *.csproj .
RUN dotnet restore
COPY . .
RUN dotnet publish --configuration Debug -o /app

FROM mcr.microsoft.com/dotnet/runtime:8.0-windowsservercore-ltsc2019
WORKDIR /app
COPY --from=build /app .
USER ContainerUser
ENTRYPOINT ["dotnet", "Shutdown.dll"]

I also tested the following variables:

Microsoft.Extensions.Hosting version 8.0.1 vs. 9.0.4 (no change)
.NET 8 vs. .NET 9 SDK and TargetFramework (no change)
Release vs. Debug publish configuration (no change)
windowsservercore-ltsc2025 images (no change)

/cc @jkotas - should we transfer this to the runtime repo for further investigation?

Transferred to runtime since this might be a bug with Microsoft.Extensions.Hosting.

Thank you too for taking this up too..
Introducing myself, I worked with @SaurabhSunil on identifying this issue and making a repro

I suspect this is the same Windows Service shutdown bug being discussed in #83093

Can you try the workaround we mention there #83093 (comment) - that can help us understand if this is the same bug.

    using var registration = PosixSignalRegistration.Create(PosixSignal.SIGTERM, (context) => context.Cancel = true);

Delete runtime build-in handling for SIGTERM

@jkotas

@jkotas which .NET version will have this fix? what windows container image tag should we use?

@ericstj

I have fixed #83093 that @ericstj pointed above. The fix will ship in .NET 10 Preview 5.

After re-reading the conversation above, I am not convinced that this fix will resolve your issue given that workaround did not work.

I will keep this issue open.

I took a look at your repro:

It is not a Windows Service. It is not affected by the problem tracked by CoreCLR's CNTRL_SHUTDOWN_EVENT handler prevents graceful exit of services during system shutdown #83093
Is uses UseConsoleLifetime and the termination signal is delivered properly

The problem seems to be that the signal handler registered by UseConsoleLifetime schedules shutdown asynchronously, returns back to Windows immediately, and Windows continues with container shutdown. I am not sure whether it is expected Windows behavior or a Windows bug.

Your workaround that registers signal handler and calls host.StopAsync().GetAwaiter().GetResult(); synchronously inside it prevents the signal handler from returning to Windows, and thus blocks Windows from continuing the container shutdown.

it does not complete when StopAsync implementation takes more than 4 second.

This sounds like shutdown grace period. It is a timeout imposed by container orchestrator. It can be configured, but the details how to configure it depend on the environment: https://www.bing.com/search?q=how%20to%20configure%20container%20shutdown%20grace%20period&qs=n&form=QBRE&sp=-1&ghc=2&lq=0&pq=how%20to%20configure%20container%20shutdown%20grace%20period&sc=12-48&sk=&cvid=BA746C3DBA0C4F49B6DA7668B00E6999

@eerhardt

This is related to #56057, cc @eerhardt

The problem seems to be that the signal handler registered by UseConsoleLifetime schedules shutdown asynchronously, returns back to Windows immediately, and Windows continues with container shutdown. I am not sure whether it is expected Windows behavior or a Windows bug.

ConsoleLifetime calls StopApplication

Which we document as blocking until stop is complete.

Signals the ApplicationStopping event and blocks until it completes.

The main thread kicks things off with StartAsync, then blocks on WaitForShutdownAsync() until ApplicationStopping is signaled.

runtime/src/libraries/Microsoft.Extensions.Hosting.Abstractions/src/HostingAbstractionsHostExtensions.cs

Lines 90 to 116 in 9231f34

    
                   public static async Task WaitForShutdownAsync(this IHost host, CancellationToken token = default) 
        
                   { 
        
                       IHostApplicationLifetime applicationLifetime = host.Services.GetRequiredService<IHostApplicationLifetime>(); 
        
                       token.Register(state => 
        
                       { 
        
                           ((IHostApplicationLifetime)state!).StopApplication(); 
        
                       }, 
        
                       applicationLifetime); 
        
           #if NET8_0_OR_GREATER 
        
                       await Task.Delay(Timeout.Infinite, applicationLifetime.ApplicationStopping).ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing); 
        
           #else 
        
                       var waitForStop = new TaskCompletionSource<object?>(TaskCreationOptions.RunContinuationsAsynchronously); 
        
                       applicationLifetime.ApplicationStopping.Register(obj => 
        
                       { 
        
                           var tcs = (TaskCompletionSource<object?>)obj!; 
        
                           tcs.TrySetResult(null); 
        
                       }, waitForStop); 
        
                       await waitForStop.Task.ConfigureAwait(false); 
        
           #endif 
        
                       // Host will use its default ShutdownTimeout if none is specified. 
        
                       // The cancellation token may have been triggered to unblock waitForStop. Don't pass it here because that would trigger an abortive shutdown. 
        
                       await host.StopAsync(CancellationToken.None).ConfigureAwait(false); 
        
                   }

AFAICT it looks to me like a race to run StopAsync after the Posix handler signals ApplicationStopping and before the process is forcibly killed by OS.

The docs for CNTRL_SHUTDOWN_EVENT do indicate that the process will be killed after returning from this -- https://learn.microsoft.com/en-us/windows/console/handlerroutine#remarks

Return FALSE. If none of the registered handler functions returns TRUE, the default handler terminates the process.
Return TRUE. In this case, no other handler functions are called and the system terminates the process.

In this case - after calling other handlers, since the result was cancelled.

This seems different than Linux where SIGTERM can be cancelled/ignored allowing the process to continue to run.

It looks to me like before #56057 the ConsoleLifetime would have blocked the OnProcessExit handler until it was disposed, which would only happen after host was shutdown. After that change it may have introduced this race.

I wonder if we should block the completion of CNTRL_SHUTDOWN_EVENT (AKA PosixSignal.SIGTERM) on Windows to allow the app to run to completion?

In case it helps, when I worked on application shutdown, I wrote this which describes my understanding of the shutdown process:

https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.Extensions.Hosting/docs/HostShutdown.md

I wonder if we should block the completion of CNTRL_SHUTDOWN_EVENT (AKA PosixSignal.SIGTERM) on Windows to allow the app to run to completion?

The puzzling part is that this works just fine on Windows Nano. Returning from the signal handler on Windows Nano works like signal handlers on Unix - it won't terminate the process. It may be a good idea to get a confirmation from Windows that blocking in the signal handler would be the correct fix.

@eerhardt

I looked into this and found the root cause of the issue. Windows has two different implementations of process shutdown. One is in winsrv.dll and the other in winsrvext.dll. The former is used for nano and it does not kill the process after returning from it's control handler, instead it waits for the process to exit on its own (given some timeout). The latter is used on servercore (and likely most other SKUs) - it will kill the process as soon as it returns from the control handler if the process has not exited on its own as mentioned in the docs. I am discussing this with Windows folks about documenting this better and potentially improving - though I'm sure this sort of thing is mired in perf and compatibility.

I think we need to work better for the winsrvext.dll case regardless - since this exists today and is publicly documented behavior.

That means we need to clean up as much as possible by the time our control handler returns. Ideally we don't even return at all but let the process exit while in the handler. That way there is no race condition where user code in main (which we cannot wait on) might run.

I attempted to implement this, by just blocking the thread in

runtime/src/libraries/Microsoft.Extensions.Hosting/src/Internal/ConsoleLifetime.netcoreapp.cs

Lines 27 to 33 in d1e4319

    
           private void HandlePosixSignal(PosixSignalContext context) 
        
           { 
        
               Debug.Assert(context.Signal == PosixSignal.SIGINT || context.Signal == PosixSignal.SIGQUIT || context.Signal == PosixSignal.SIGTERM); 
        
               context.Cancel = true; 
        
               ApplicationLifetime.StopApplication(); 
        
           }

Which allows main to complete and exit the process.

This would deadlock, since main tries to dispose of the ConsoleLifetime, which in turn disposes the signal registrations in UnregisterShutdownHandlers. Windows itself will block in the call to SetConsoleCtrlHandler if any handlers are currently running. I think the solution here is to just leak those registrations. If we're in shutdown we know the process is going away - no need to unregister the handlers if dispose is called (we can't possibly do it anyway, since we can't call SetConsoleCtrlHandler while the handler is running and the process will be killed immediately upon return).

I'll propose a PR to fix this and try to think through how I might test it. I'll also update your docs, @eerhardt.

I haven't forgotten about this - still plan to get a fix in.

Navigation Menu

Repository navigation

IHostedService.StopAsync not invoked with `8.0-windowsservercore-ltsc2019` image #115206

Describe the bug

Which .NET image(s) are you using?

Steps to reproduce

Other information

Output of `docker version`

Output of `docker info`

4 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	public static async Task WaitForShutdownAsync(this IHost host, CancellationToken token = default)
	{
	IHostApplicationLifetime applicationLifetime = host.Services.GetRequiredService<IHostApplicationLifetime>();

	token.Register(state =>
	{
	((IHostApplicationLifetime)state!).StopApplication();
	},
	applicationLifetime);

	#if NET8_0_OR_GREATER
	await Task.Delay(Timeout.Infinite, applicationLifetime.ApplicationStopping).ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing);
	#else
	var waitForStop = new TaskCompletionSource<object?>(TaskCreationOptions.RunContinuationsAsynchronously);
	applicationLifetime.ApplicationStopping.Register(obj =>
	{
	var tcs = (TaskCompletionSource<object?>)obj!;
	tcs.TrySetResult(null);
	}, waitForStop);

	await waitForStop.Task.ConfigureAwait(false);
	#endif

	// Host will use its default ShutdownTimeout if none is specified.
	// The cancellation token may have been triggered to unblock waitForStop. Don't pass it here because that would trigger an abortive shutdown.
	await host.StopAsync(CancellationToken.None).ConfigureAwait(false);
	}

	private void HandlePosixSignal(PosixSignalContext context)
	{
	Debug.Assert(context.Signal == PosixSignal.SIGINT \|\| context.Signal == PosixSignal.SIGQUIT \|\| context.Signal == PosixSignal.SIGTERM);

	context.Cancel = true;
	ApplicationLifetime.StopApplication();
	}

IHostedService.StopAsync not invoked with 8.0-windowsservercore-ltsc2019 image #115206

Description

Describe the bug

Which .NET image(s) are you using?

Steps to reproduce

Other information

Output of docker version

Output of docker info

Activity

lbussell commented on Apr 30, 2025

lbussell commented on Apr 30, 2025

iJinsPeter commented on Apr 30, 2025

ericstj commented on May 5, 2025

4 remaining items

iJinsPeter commented on May 16, 2025

jkotas commented on May 16, 2025

jkotas commented on May 16, 2025

ericstj commented on May 19, 2025

eerhardt commented on May 19, 2025

jkotas commented on May 19, 2025

ericstj commented on May 23, 2025

ericstj commented on Jun 3, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

IHostedService.StopAsync not invoked with `8.0-windowsservercore-ltsc2019` image #115206

Output of `docker version`

Output of `docker info`