Micro-accuracy for Multiclass Classification tests

Micro-accuracy is generally better aligned with the business needs of ML predictions. If we are only choosing one metric to report for a multiclass classification task, it should be micro-accuracy.

Example, for a support ticket classification task: (maps incoming tickets to teams)

Micro-accuracy -- how often does an incoming ticket get classified to the right team?
Macro-accuracy -- for an average team, how often is an incoming ticket correct for their team?

Macro-accuracy overweights small teams in this example; a small team which gets only 10 tickets per year counts as much as a large team with 10k tickets per year. Micro-accuracy in this case correlates better with the business need of, "how much time/money can the company save by automating my ticket routing process".

Below we are reporting only macro-accuracy:

machinelearning/test/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs

Lines 42 to 43 in 23659b0

    
           nameof(ClassificationMetrics.AccuracyMacro), 
        
           _metrics.AccuracyMacro.ToString("0.##", CultureInfo.InvariantCulture));

Benchmark output: (src)

              Method |         Mean |      Error |     StdDev |        Extra Metric |
-------------------- |-------------:|-----------:|-----------:|--------------------:|
         PredictIris |     1.650 ms |  0.0151 ms |  0.0141 ms | AccuracyMacro: 0.98 |
 PredictIrisBatchOf1 |     1.599 ms |  0.0362 ms |  0.0339 ms | AccuracyMacro: 0.98 |
 PredictIrisBatchOf2 |     1.646 ms |  0.0179 ms |  0.0167 ms | AccuracyMacro: 0.98 |
 PredictIrisBatchOf5 |     1.635 ms |  0.0192 ms |  0.0179 ms | AccuracyMacro: 0.98 |

For this Iris dataset benchmark, the difference between macro/micro-accuracy are non-important, though it sets a bad precedent which will be replicated in further benchmarks.

Work:

The above test should be changed to report micro-accuracy
See if other benchmarks are reporting macro-accuracy
(future) Report additional metrics instead of just one

I'll take a shot at this one.

I'm having trouble getting the benchmarks to run. I follow the steps in Readme.md and I get an exception

// ************************** // Benchmark: StochasticDualCoordinateAscentClassifierBench.TrainIris: Job-EJJOGT(BuildConfiguration=Release, Toolchain=netcoreapp2.1, MaxIterationCount=20, WarmupCount=1) // *** Execute *** // Launch: 1 / 1 // BeforeAnythingElse // Benchmark Process Environment Information: // Runtime=.NET Core 2.1.5 (CoreCLR 4.6.26919.02, CoreFX 4.6.26919.02), 64bit RyuJIT // GC=Concurrent Workstation // Job: Job-RVQKPR(BuildConfiguration=Release, MaxIterationCount=20, WarmupCount=1) OverheadJitting 1: 1 op, 400228.15 ns, 400.2281 us/op System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.ML.PipelineInference, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, IntPtr ptrLoadContextBinder) at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, IntPtr ptrLoadContextBinder) at System.Reflection.Assembly.Load(AssemblyName assemblyRef) at Microsoft.ML.Runtime.AssemblyRegistration.LoadStandardAssemblies() in C:\Users\Sebastian\Downloads\machinelearning-master\src\Microsoft.ML.Legacy\AssemblyRegistration.cs:line 60 at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor) at System.Lazy1.CreateValue()
at Microsoft.ML.Runtime.AssemblyRegistration.RegisterAssemblies(IHostEnvironment environment) in C:\Users\Sebastian\Downloads\machinelearning-master\src\Microsoft.ML.Legacy\AssemblyRegistration.cs:line 30
at Microsoft.ML.Runtime.Experiment..ctor(IHostEnvironment env) in C:\Users\Sebastian\Downloads\machinelearning-master\src\Microsoft.ML.Legacy\Runtime\Experiment\Experiment.cs:line 42
at Microsoft.ML.Legacy.LearningPipeline.TrainTInput,TOutput in C:\Users\Sebastian\Downloads\machinelearning-master\src\Microsoft.ML.Legacy\LearningPipeline.cs:line 167
at Microsoft.ML.Benchmarks.StochasticDualCoordinateAscentClassifierBench.Train(String dataPath) in C:\Users\Sebastian\Downloads\machinelearning-master\test\Microsoft.ML.Benchmarks\StochasticDualCoordinateAscentClassifierBench.cs:line 58
at BenchmarkDotNet.Autogenerated.Runnable_0.WorkloadActionNoUnroll(Int64 invokeCount) in C:\Users\Sebastian\Downloads\machinelearning-master\bin\AnyCPU.Release\Microsoft.ML.Benchmarks\netcoreapp2.1\e32fb988-4165-4058-83bd-d2834e52485b\e32fb988-4165-4058-83bd-d2834e52485b.notcs:line 474
at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost host) in C:\Users\Sebastian\Downloads\machinelearning-master\bin\AnyCPU.Release\Microsoft.ML.Benchmarks\netcoreapp2.1\e32fb988-4165-4058-83bd-d2834e52485b\e32fb988-4165-4058-83bd-d2834e52485b.notcs:line 151
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\Users\Sebastian\Downloads\machinelearning-master\bin\AnyCPU.Release\Microsoft.ML.Benchmarks\netcoreapp2.1\e32fb988-4165-4058-83bd-d2834e52485b\e32fb988-4165-4058-83bd-d2834e52485b.notcs:line 65
// AfterAll
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!`

Could you please give me a hint?

Hey Sebastian, could you tell us what code is giving you this error?

Hi Artidoro, thanks for having a look at this. I downloaded the repository and then tried to run the benchmarks as described in Readme.md i.e.

build.cmd -release -buildNative
build.cmd -- /t:DownloadExternalTestFiles /p:IncludeBenchmarkData=true
dotnet run -c Release

Does that answer your question?

Sorry I forgot to mention that the error only occurs after I choose benchmark 8 StochasticDualCoordinateAscentClassifierBench. Other benchmarks (e.g. benchmark 7) run without problems.

The assembly Microsoft.ML.PipelineInference is not contained in the build which is produced after starting the benchmark. This seems to be the reason why I get the exception above. Any ideas how to fix this?

Shouldn't we be reporting both to align with most current ML libraries and report Macro/Micro metrics? In the Azure ML SDK both types are reported.

Navigation Menu

Repository navigation

Micro-accuracy for Multiclass Classification tests #1268

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	nameof(ClassificationMetrics.AccuracyMacro),
	_metrics.AccuracyMacro.ToString("0.##", CultureInfo.InvariantCulture));

Micro-accuracy for Multiclass Classification tests #1268

Description

Activity

SebHel commented on Oct 20, 2018

SebHel commented on Oct 25, 2018

artidoro commented on Oct 27, 2018

SebHel commented on Oct 28, 2018

SebHel commented on Oct 29, 2018

SebHel commented on Nov 2, 2018

bartczernicki commented on Feb 26, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions