Improper/missing IAsyncEnumerator disposal?

[InspiringCode]

Excel Type

• XLSX

MiniExcel Version

1.42.0

Description

When calling SaveAsAsync multiple times sequentially, it seems that the IAsyncEnumerator is not disposed correctly. I have the following code:

List<IAsyncEnumerable<T>> partitions = [];

int i = 0;
while (i < totalAmountExcelFiles)
{
    partitions.Add(query.Skip(i * maxRowsPerFile).Take(maxRowsPerFile).AsAsyncEnumerable());
    i++;
}

int j = 1;
using var zip = new ZipArchive(target, ZipArchiveMode.Create, leaveOpen: true);
foreach (IAsyncEnumerable<T> partition in partitions)
{
    var entry = zip.CreateEntry($"{fileNameTemplate}_{j++:D2}.xlsx", CompressionLevel.Optimal);

    await using (Stream entryStream = entry.Open())
    {
        await MiniExcel.SaveAsAsync(entryStream, partition, cancellationToken: ct);
        await entryStream.FlushAsync(ct);
    }
}

When executing it throws an Npgsql.NpgsqlOperationInProgressException ("(0x80004005): A command is already in progress") on the second iteration with the callstack:

   at Npgsql.ThrowHelper.ThrowNpgsqlOperationInProgressException(NpgsqlCommand command)
   at Npgsql.Internal.NpgsqlConnector.<StartUserAction>g__DoStartUserAction|282_0(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation)
   at Npgsql.Internal.NpgsqlConnector.StartUserAction(ConnectorState newState, NpgsqlCommand command, CancellationToken cancellationToken, Boolean attemptPgCancellation)
   at Npgsql.NpgsqlCommand.<ExecuteReader>d__120.MoveNext()
   at Npgsql.NpgsqlCommand.<ExecuteReader>d__120.MoveNext()
   at Npgsql.NpgsqlCommand.<ExecuteDbDataReaderAsync>d__113.MoveNext()
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.<ExecuteReaderAsync>d__18.MoveNext()
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.<ExecuteReaderAsync>d__18.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.<InitializeReaderAsync>d__21.MoveNext()
   at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.<ExecuteAsync>d__7`2.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.<MoveNextAsync>d__20.MoveNext()
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at MiniExcelLibs.WriteAdapter.AsyncEnumerableWriteAdapter`1.<GetRowsAsync>d__6.MoveNext()
   at MiniExcelLibs.WriteAdapter.AsyncEnumerableWriteAdapter`1.<GetRowsAsync>d__6.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at MiniExcelLibs.OpenXml.ExcelOpenXmlSheetWriter.<WriteValuesAsync>d__7.MoveNext()
   at MiniExcelLibs.OpenXml.ExcelOpenXmlSheetWriter.<WriteValuesAsync>d__7.MoveNext()
   at MiniExcelLibs.OpenXml.ExcelOpenXmlSheetWriter.<CreateSheetXmlAsync>d__3.MoveNext()
   at MiniExcelLibs.OpenXml.ExcelOpenXmlSheetWriter.<SaveAsAsync>d__0.MoveNext()
   at MiniExcelLibs.MiniExcel.<SaveAsAsync>d__3.MoveNext()

If I change the loop to a simple async enumeration:

int debug = 1;
foreach (IAsyncEnumerable<T> partition in partitions)
{
    var entry = zip.CreateEntry($"{fileNameTemplate}_{j++:D2}.xlsx", CompressionLevel.Optimal);

    await foreach (T item in partition)
        debug++;
}

it works without problems, so it seems that MiniExcel doesn't correctly dispose the IAsyncEnumerator.

[izanhzh]

@InspiringCode I guess this issue is caused by the limitations of the EF Core's DbContext. EF Core's DbContext is not thread-safe. When multiple queries are executed concurrently, it can lead to conflicts like the NpgsqlOperationInProgressException you encountered.

[michelebastione]

@izanhzh That was my thought as well but then again from the sample provided it appears that the calls are not being parallelized, so if all IAsyncEnumerables are awaited sequentially it shouldn't matter. There might actually be something fishy going on, I'll investigate.

[izanhzh]

@izanhzh That was my thought as well but then again from the sample provided it appears that the calls are not being parallelized, so if all IAsyncEnumerables are awaited sequentially it shouldn't matter. There might actually be something fishy going on, I'll investigate.

You're absolutely right—I hadn't fully grasped the issue initially. I've managed to pinpoint the problem: it might be that the  GetAsyncEnumerator  in  AsyncEnumerableWriteAdapter  isn't being properly disposed of.

[michelebastione]

Even disposing that still doesn't make it work as expected, but maybe we can get rid of it entirely. I've done a couple of tries and it seems it might work, but this is a non trivial integration testing problem so I still need to make sure of it.