Windows service infrequently results in error 1067 on stop operation

Description

.NET 6 published application sometimes crashes on service stop operation. I created a web application from the template and only added the Microsoft.Extensions.Hosting.WindowsServices (version 6.0.0) package and called UseWindowsService() to make sure the application listens for service controller requests.

Here is the code:

using Microsoft.Extensions.Hosting.WindowsServices;

var options = new WebApplicationOptions
{
    Args = args,
    ContentRootPath = WindowsServiceHelpers.IsWindowsService() ? AppContext.BaseDirectory : default
};

var builder = WebApplication.CreateBuilder(options);

// Add services to the container.
builder.Services.AddRazorPages();
builder.Host.UseWindowsService();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
}

app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapRazorPages();

app.Run();

Reproduction Steps

Repository with code, script and repro steps: https://github.com/zryska/net6-repro

Create a new Web application using a template
Make the changes to add UseWindowsService() call as shown above
Publish the application as SFD or FDD for win-x64 runtime
Create a new service using sc create command
Start the service and verify it's running
Stop the service
Start the service
Repeat steps 4 and 5 until the issue is reproduced

It can be done through the services UI or using a script. On my system it usually happens 1 time out of 10 graceful shutdowns - but it isn't deterministic.

When done through services UI the following will be displayed:

Even log => Windows Logs -> System will contain the following message (when done through script or UI):

The .NET6 test service service terminated unexpectedly.  It has done this 26 time(s).

Expected behavior

Service stops gracefully every time.

Actual behavior

Non-graceful shutdowns are happening.

Regression?

No response

Known Workarounds

Explicitly assigning value to ServiceName fixes the problem.

builder.Host.UseWindowsService(o => o.ServiceName = "AnyServiceName");

Configuration

.NET version: .NET 6.0.100
OS version: Windows 10 pro build 19044.1348
Architecture: x64

Other information

No response

Any chance you can get more information here to narrow down where the problem is? Maybe try getting a crash dump.

https://github.com/dotnet/diagnostics/blob/main/documentation/dotnet-dump-instructions.md

Or adding a try-catch somewhere in your project to try to catch the exception.

Can you attach a Debugger before stopping the service? If you do, does it still crash?

I wasn't able to get a crash dump generated for some reason. Also when I included try-catch block around the main method it didn't catch anything. The application seems to be crashing after it leaves the main method. When I run the same code on Linux just with systemd package instead of WindowsServices the same code works just fine.

Your best bet here is to get more information - what is the stacktrace / message / etc that is causing the process to crash.

The issue is not surfacing when I have debugger attached. Would you be able to give it a try on your end by following the repro steps or is there something else I could provide in terms of repro to make it easier?

I wasn't able to get a crash dump generated for some reason

Was dotnet-dump not working for you? What was the reason?

My understanding was that dotnet-dump allows only to generate a dump at time of executing the collect command. I was using SysInternals ProcDump but it didn't create any crash dump even when service manager reported crash.

This is procdump output from event when service controller reported ungraceful shutdown:

PS C:\Users\zryska\Tools\Procdump> .\procdump64.exe -ma -e WebApplication1.exe

ProcDump v10.11 - Sysinternals process dump utility
Copyright (C) 2009-2021 Mark Russinovich and Andrew Richards
Sysinternals - www.sysinternals.com

Process:               WebApplication1.exe (41180)
Process image:         C:\Program Files\Test\WebApplication1.exe
CPU threshold:         n/a
Performance counter:   n/a
Commit threshold:      n/a
Threshold seconds:     n/a
Hung window check:     Disabled
Log debug strings:     Disabled
Exception monitor:     Unhandled
Exception filter:      [Includes]
                       *
                       [Excludes]
Terminate monitor:     Disabled
Cloning type:          Disabled
Concurrent limit:      n/a
Avoid outage:          n/a
Number of dumps:       1
Dump folder:           C:\Users\zryska\Downloads\Procdump\
Dump filename/mask:    PROCESSNAME_YYMMDD_HHMMSS
Queue to WER:          Disabled
Kill after dump:       Disabled


Press Ctrl-C to end monitoring without terminating the process.

[22:27:46] Exception: E0434352.CLR
[22:27:46] Exception: E0434352.CLR
[22:27:46] Exception: E0434352.CLR
[22:27:46] Exception: E0434352.CLR
[22:27:46] Exception: 00000006
[22:27:46] Exception: E0434352.CLR
[22:27:46] Exception: E0434352.CLR
[22:27:47] The process has exited.
[22:27:47] Dump count not reached.

C:\Users\zryska\Downloads\Procdump\

There's no dump files in that folder?

Can you try https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps? Or possibly https://docs.microsoft.com/en-us/visualstudio/debugger/debug-using-the-just-in-time-debugger?view=vs-2022#BKMK_Enabling?

No. No dumps were created. I should also mention that the main method apparently exists gracefully since in case I add a line of code to create a file before exiting main the file gets created even when the service control manager reports that the application crashed. I was trying the registry approach in the past when I was suspecting that something is wrong in our code application without results but I can give it a try again with the empty WebApplication project.

Look at the event log to se if there's more information.

Hi David, unfortunately the event log doesn't contain anything actionable. The alleged crash event is logged in the system log and there is nothing in the Application log around that time.

Here is the raw event (removed computer name) in case there is something I missed:

 - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" /> 
  <EventID Qualifiers="49152">7034</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x8080000000000000</Keywords> 
  <TimeCreated SystemTime="2021-12-10T08:15:28.6705360Z" /> 
  <EventRecordID>105858</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="1632" ThreadID="29084" /> 
  <Channel>System</Channel> 
  <Computer>ComputerName</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data Name="param1">.NET6 test service</Data> 
  <Data Name="param2">7</Data> 
  <Binary>2E004E0045005400360020007400650073007400200073006500720076006900630065000000</Binary> 
  </EventData>
  </Event>

I tried to replicate it on another machine with lower HW specs:
CPU: 4 cores
RAM: 8 GB
OS: Windows 10 Enterprise build: 19042.1348

It still happened but much less frequently. I let the script run for a while and out of 1000 restarts there were just two "crash" events reported in the event log.

For comparison my development machine HW specs:
CPU: 16 cores
RAM: 32 GB
OS: Windows 10 Pro build: 19044.1348

I moved the repro code into a separate repository if it helps: https://github.com/zryska/net6-repro

@zryska

@zryska use procdump64.exe -mp -t -e 1 -x . "WebApplication1.exe"

@buyaa-n

We should prioritize the fix for this and consider backporting to .NET 7.0 and 6.0. The workaround to wait some time before exiting doesn't sufficiently solve the problem, because it doesn't hold up the progress of

runtime/src/libraries/Microsoft.Extensions.Hosting.Abstractions/src/HostingAbstractionsHostExtensions.cs

Line 75 in 967250c

await asyncDisposable.DisposeAsync().ConfigureAwait(false);

Which will dispose the service and make it impossible for the service state to be updated.

@buyaa-n as we discussed, we can try the fix of signaling an in Run when ServiceBase.Run(..) returns, then waiting on this in StopAsync. This will ensure that the service has safely exited.

Keeping open for servicing.

@ericstj

@ericstj would be awesome if this would be backported to .NET 6 and 7. We are currently experiencing this exact problem with one of our applications.

Yes, that’s the plan. Most likely the June release.

If you have the ability to test daily bits you can try the net8.0 package to confirm the fix works for you. That can be restored from

runtime/NuGet.config

Line 22 in f107b4b

    
           <add key="dotnet8" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json" />

If you are able to try and confirm the fix that helps us ensure we got things right. Thanks!

@ericstj

@ericstj I tried the fix on behalf of @MD-V and so far everything seems to work properly.

Navigation Menu

Repository navigation

Windows service infrequently results in error 1067 on stop operation #62579

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

27 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Windows service infrequently results in error 1067 on stop operation #62579

Description

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

Activity

eerhardt commented on Dec 9, 2021

zryska commented on Dec 9, 2021

eerhardt commented on Dec 9, 2021

zryska commented on Dec 9, 2021

eerhardt commented on Dec 9, 2021

zryska commented on Dec 9, 2021

eerhardt commented on Dec 9, 2021

zryska commented on Dec 9, 2021

davidfowl commented on Dec 10, 2021

zryska commented on Dec 10, 2021

zryska commented on Dec 10, 2021

OwnageIsMagic commented on Dec 12, 2021

27 remaining items

ericstj commented on Mar 24, 2023

ericstj commented on Apr 6, 2023

MD-V commented on Apr 21, 2023

ericstj commented on Apr 21, 2023

FlorianAlbert commented on Apr 27, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions