Repository navigation
Let ML.NET cancel unmanaged code. #6465
Description
System Information:
- OS & Version: Windows 11 [Version 10.0.22621.819]
- ML.NET Version: ML.NET v2.0.0
- .NET Version: .NET 7.0.100
Describe the bug
I run an ML.NET experiment for 10 hours. I limited the training for FastTree and FastForest algorithms and I made sure that every new best model is saved even if the whole run fails. I locked Windows and let it run during the night.
When I checked it in the morning (12 hours after the experiment started) I couldn't even log in, I had to do a power off-power on to get access. When I checked the logs it looked like the last experiment was saved after ~5 hours of running.
The dataset wasn't particularly big this time, it was only a 205 MB CSV file. My system has 32 GB RAM, and I set the maximum memory usage for the experiment to 20 GB.
To Reproduce
Steps to reproduce the behavior:
- Create a pipeline that includes only FastTree and FastForest, for example:
var pipeline =
mlContext.Auto().Featurizer(trainTestData.TrainSet, numericColumns: new[] { "Features" })
.Append(mlContext.Auto().Regression(useFastTree: true, useLbfgs: false, useSdca: false, useFastForest: true, useLgbm: false));
- Create an ML.NET experiment and set its memory limit. I set it to 20 GB, because I usually have ~24 GB available when I start the training.
var experiment = mlContext.Auto().CreateExperiment();
experiment
.SetPipeline(pipeline)
.SetTrainingTimeInSeconds(trainingTimeInSeconds)
.SetRegressionMetric(RegressionMetric.RSquared, labelColumn: "Label")
.SetDataset(trainTestData.TrainSet, trainTestData.TestSet)
.SetMonitor(monitor)
.SetMaximumMemoryUsageInMegaByte(20 * 1024);
- Run the experiment for at least 10 hours
- Check if you can still use the system after 5+ hours of running
Expected behavior
I would expect the experiment to run without a system-freezing halt.
Screenshots, Code, Sample Projects
The 205 MB CSV dataset can be downloaded from here:
BBD_20221122__TrainingData.Sleep.MLP12__MLP12_0p25Hz-250Hz__Session_SegmentedData_Sleep_Level__15782rows.zip
Additional context
My project is open-source so both the source and the data are available. Let me know if you need them for testing.
Activity
andrasfuchs commentedon Nov 26, 2022
I run further tests, and I think I found the root cause of this problem: the LightGbmRegression estimator can't be cancelled during it's startup and it crashes if the systems runs out of virtual memory.
Here are my logs:
I set the cancellation thresholds to
Please notice that the estimator was cancelled successfully in trials 4, 6, 9, 12,13 and 17. If you look at the last trial (20) you can see that my monitor tries to cancel the estimator again and again without success until the process runs out of virtual memory and it gets the "bad allocation" exception repeatedly and finally crashes the whole process.
michaelgsharp commentedon Nov 28, 2022
@LittleLittleCloud any thoughts on this? looks like a cancellation error. Unless its a LightGBM issue itself.
LittleLittleCloud commentedon Nov 29, 2022
LightGBM can't be canceled when it's calling into unmanaged dll because cancellation pattern of ml.net is not implemented there. @michaelgsharp is there a way to cancel a running estimator when it's running into unmanaged code by calling
mlcontext.CancelExection()?andrasfuchs commentedon Dec 2, 2022
It might a little drastic, but I will try to do some experiments when I forcefully terminate the thread that runs the unmanaged code in my
IMonitorevent handler. For this I will need to modify the ML.NET part to pass the unmanaged ThreadId in theTrialSettingsclass, but if it works it would be still better than crashing the whole process because I run out of free virtual memory.LittleLittleCloud commentedon Dec 2, 2022
@andrasfuchs Another solution that might be an easier path for you is to use a smaller lightGBM search space. The current default lightGBM search space might be too large to an extend that doesn't actually benefit the final training result.
Here's how to pass a smaller lightGBM search space, which we also apply when running azure benchmark and proves to have little impact to final result
michaelgsharp commentedon Dec 5, 2022
@LittleLittleCloud looks like the work item for this should be to add that capability when we are calling into unmanaged code.
[-]Long running AutoML experiment freezes Windows[/-][+]Let ML.NET cancel unmanaged code.[/+]andrasfuchs commentedon Dec 6, 2022
@LittleLittleCloud Thank you for the suggestion, and sorry for not getting back to you sooner, I just wanted to make sure that I test the reduced search space correctly.
It seems to be working fine: I could run 100+ LightGBM trials without any issues. The memory usage topped at 12.4 GB compared to the 46.5+ GB I had previously. I'll still do some test this week with long-running training sessions, but it looks like a great temporary solution!
I include my source code and logs for reference.
andrasfuchs commentedon Dec 6, 2022
Update: @LittleLittleCloud It turns out it's not bulletproof, my experiment just crashed my system again after ~2 hours of running. I agree with @michaelgsharp, cancelling unmanaged code would be the ideal solution.
I need to disable LightGBM for now.
andrasfuchs commentedon Dec 7, 2022
The cancellation mechanism seems to be unreliable for FastForestRegression as well. I run a ~11 hour-long experiment, and if you look at the logs below you'll see that the cancellation of trial 741 took more than 1.5 minutes!
The first cancellation request issued at 08:48:54, but it was only completed at 08:50:27. Would it be possible to apply a more aggressive task/thread cancellation so that we can stop a trial instantly?
LittleLittleCloud commentedon Dec 7, 2022
@andrasfuchs
For
FastForestRegressionwe can always improve cancellation experience by checking ifMLContextis alive or not as long as it's running in managed code.For unmanaged code, the cancellation situation will be a bit more complex and might be out of ml.net's scope depends on if we own that unmanaged code. @michaelgsharp do you have any idea if there's an easier way to implement cancellation pattern in lightGBM unmanaged training part?
andrasfuchs commentedon Dec 22, 2022
I understand if you are hesitant to implement a drastic unmanaged code termination in ML.NET, but could you pass the identifier of the thread that runs the trial (including the unmanaged code) to the IMonitor?
Then the user could decide if they wanted to risk a thread termination or let the thread run even if it hits a performance metric threshold.
My experience is that it would be still better to terminate the thread if we are running out of virtual memory, because the alternative is a system-wide crash that affects other running processes as well.
LittleLittleCloud commentedon Dec 22, 2022
@andrasfuchs
Just to see if I understand your suggestion correctly. Are you talking about just terminating the thread of current running trial, versus terminating the thread of the entire AutoMLExperiment.
The two kinds of terminating will lead to different solutions. If you just want to terminate the entire AutoMLExperiment if it hits a performance threshold, I would suggest moving this termination logic to your code base by making sure to start AutoMLExperiment in a separate thread and monitoring system memory usage in order to terminate AutoMLExperiment thread if it's using too much memory.
If you want to just terminate the current running trial without fail the entire AutoMLExperiment training, we probably need to do more than just pass the current trial's thread id to monitor for later termination. Because AutoMLExperiment kicks off trials using
Task, which could be running on a thread that is shared with multiple other tasks and call termination might cause unexpected behavior.Overall, I feel like the dilemma can be summarized to perform cancellation or termination over a task that might not respond or doesn't implement a cancellation pattern. @tarekgh @michaelgsharp might know better on this dilemma and I can dig into it to see if there's existing current solutions.
andrasfuchs commentedon Dec 22, 2022
@LittleLittleCloud
Thank you for your explanation, I didn't know these properties of the AutoMLExperiment, it makes sense.
Terminating the whole AutoMLExperiment wouldn't be ideal, but at least it would protect the system from crashing.
I'm not totally sure about this, but if I remember correctly forcefully terminating a Task isn't supported, only Threads can be aborted/terminated.
I assumed mistakenly that each trial has its own thread (with its managed and unmanaged sub-threads, and tasks) so it's possible to abort just that without killing the experiment if the normal cancellation technique fails.
It is a dilemma indeed, and I'm interested in your findings!
LittleLittleCloud commentedon Dec 28, 2022
That's right, although there's hacky way to
killa task, it's nearly always be highly unrecommended. According to this discussion,Taskshould not be used for abortive cancellation purpose. And it's recommended to either use aprocessorthreadwhen it's necessary to call into third-party code where cancellation pattern might not be well-implemented.So to resolve this issue, I can see two work-item here: 1) improving LightGbm cancellation experience by running the native code part in a separate thread or process so it can be terminated safely when there's no response and 2) In AutoML, use
Threadto start trials