Exception in IdleComplete

[ekalchev]

I have the following setup:

1. Dedicated thread which uses ImapClient synchronous methods
2. Main Thread which cancel cancellation token passed to ImapClient.Idle(...)

I am starting ImapClient.Idle in a dedicated thread and Idle method blocks the thread. It seems because of network error an exception is thrown from inside IdleComplete method. The problem is I can't catch this exception. It somehow bubbling to my main thread and crashing the application. I still can't figure out how this end up in the main thread if ImapClient.Idle is ran inside worker thread.

If you look at the call stack there is no call from my application there. There is no way to catch this exception.

System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Security._SslStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at MailKit.Net.Imap.ImapStream.<FlushAsync>d__82.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MailKit.Net.Imap.ImapStream.Flush(CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapClient.IdleComplete(Object state)
   at System.Threading.CancellationCallbackInfo.ExecutionContextCallback(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.CancellationCallbackInfo.ExecuteCallback()
   at System.Threading.CancellationTokenSource.CancellationCallbackCoreWork(CancellationCallbackCoreWorkArguments args)
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   --- End of inner exception stack trace ---
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean throwOnFirstException)
   at System.Threading.CancellationTokenSource.LinkedTokenCancelDelegate(Object source)
   at System.Threading.CancellationCallbackInfo.ExecutionContextCallback(Object obj)
   at System.Threading.CancellationCallbackInfo.ExecuteCallback()
   at System.Threading.CancellationTokenSource.CancellationCallbackCoreWork(CancellationCallbackCoreWorkArguments args)
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   --- End of inner exception stack trace ---
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean throwOnFirstException)
   at System.Threading.CancellationTokenSource.TimerCallbackLogic(Object obj)
   at System.Threading.TimerQueueTimer.CallCallbackInContext(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()
   at System.Threading.TimerQueue.AppDomainTimerCallback()
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Security._SslStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at MailKit.Net.Imap.ImapStream.<FlushAsync>d__82.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MailKit.Net.Imap.ImapStream.Flush(CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapClient.IdleComplete(Object state)
   at System.Threading.CancellationCallbackInfo.ExecutionContextCallback(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.CancellationCallbackInfo.ExecuteCallback()
   at System.Threading.CancellationTokenSource.CancellationCallbackCoreWork(CancellationCallbackCoreWorkArguments args)
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   --- End of inner exception stack trace ---
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean throwOnFirstException)
   at System.Threading.CancellationTokenSource.LinkedTokenCancelDelegate(Object source)
   at System.Threading.CancellationCallbackInfo.ExecutionContextCallback(Object obj)
   at System.Threading.CancellationCallbackInfo.ExecuteCallback()
   at System.Threading.CancellationTokenSource.CancellationCallbackCoreWork(CancellationCallbackCoreWorkArguments args)
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
---> (Inner Exception #0) System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Security._SslStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at MailKit.Net.Imap.ImapStream.<FlushAsync>d__82.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MailKit.Net.Imap.ImapStream.Flush(CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapClient.IdleComplete(Object state)
   at System.Threading.CancellationCallbackInfo.ExecutionContextCallback(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.CancellationCallbackInfo.ExecuteCallback()
   at System.Threading.CancellationTokenSource.CancellationCallbackCoreWork(CancellationCallbackCoreWorkArguments args)
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)

[jstedfast]

The exception is being thrown on the main thread because you are cancelling the IDLE on the main thread.

[jstedfast]

You could have just done:

try {
   tokenSource.Cancel ();
} catch {
}

[ekalchev]

I did this but I wasn't 100% if this is the problem because I can't see CancellationTokenSource.Cancel() in the call stack.

While I was debugging I noticed that calling CancellationTokenSource.Cancel() on the main thread is executing IdleComplete(...) on the main thread
ctx.Engine.Stream.Write (buf, 0, buf.Length, ctx.CancellationToken);
ctx.Engine.Stream.Flush (ctx.CancellationToken);

while the Idle thread accessing ctx.Engine.Stream. Isn't this concurrency issue? I can't use Idle thread to cancel because it is blocked.

[jstedfast]

How is it a concurrency issue?

[ekalchev]

While you are in Idle the thread 1 is blocked at SslStream.Read and from thread 2
from this line you are calling SslStream.Write. I am not sure if this is ok but it might be depending of the implementation of SslStream.

Another possible issue - checking and changing the state ctx.Engine.State == ImapEngineState.Idle from IdleComplete which is ran on different thread than ImapClient.Idle

It is possible the engine to transition to another state from thread 1 after the check for ctx.Engine.State == ImapEngineState.Idle in thread 2. This the code will try to write DONE while not in idle and probably the server will disconnect ??? (I wonder if this happens with my exception above)

Here another scenario - main thread executing ctx.Engine.Stream.Flush (ctx.CancellationToken);
inside flush everythjing is ok but while the code is executing

logger.LogClient (output, 0, outputIndex); inside in ctx.Engine.Stream.Flush, you get disconnected. In thread 2 you'll transition the state to disconnected (while main thread will write the log) but on the main thread after flush you will change the state to ctx.Engine.State = ImapEngineState.Selected;

[jstedfast]

While you are in Idle the thread 1 is blocked at SslStream.Read and from thread 2
from this line you are calling SslStream.Write. I am not sure if this is ok but it might be depending of the implementation of SslStream.

That's the expected situation.

The SslStream.Read() call is almost guaranteed to be in a blocking state until the doneToken is cancelled, thereby causing the IdleComplete callback to be called to send a DONE\r\n command to the server which will cause the server to send more data (typically something along the lines of A######## OK IDLE complete).