Example showing how to use 1DS C API shared lib in .NETcore 3.x app

[Max Golovanov (maxgolov)]

C# projection layer - #4

List of changes:

• allow build.sh to accept parameters and cmake options
• clean-up CMakeLists.txt to allow building a shared library (tested on Mac only, need to test on Linux)
• update corresponding packaging scripts to include the shared library output
• update package versions from v3.2 to v3.3
• mod mat.h to pack structure
• draft example that shows how to P/Invoke from .NET Core 3.x into shared library (tested on Mac OS X and Windows). This is a DRAFT, not final. But it is a working draft to upload events to Collector.

Practical reasons for this implementation:

Reason 1. New ways to express an event for Geneva IFx SDK without using Bond. Main intent is to move IFx customers away from Bond in their code to 1DS-style EventProperties.

A concept of strongly-typed variant dictionary that can be used to express an event in a very user-friendly, intuitive manner. This is C# code that is pretty much identical to API surface we have in C++11 land and other 1DS SDKs:

                var props = new EventProperties() {
                    { "strKey", "value1" },
                    { "intKey", 12345 },
                    { "dblKey", 0.12345 } ,
                    { "guidKey", new Guid("73e21739-9d4e-497d-9c66-8e399a532ec9") }
                };

This managed object is implemented as "variant dictionary":

Dictionary<string, EventProperty>

where EventProperty - allows to contain the standard supported by collector strong types, plus optional Pii Kind tag. That managed container object then gets converted to native structure passed down to 1DS C++ SDK via "ABI stable" C API. The marshaling is not ideal at the moment. I believe we can make it x2 times faster by avoiding memcpy in a few spots.

Reason 2. Prove that it's a viable approach from performance standpoint that can be used in system services written in C#.

Benchmarking how fast we can go -- measuring combined time of the following steps:

• emit an event in managed code
• translate it from managed into native representation, including string translations from UTF-16
• serialize it with native code Common Schema 4.x Bond serializer
• save it to disk storage in SQLite DB

Preliminary results of early implementation:

• 9 seconds to process 100,000 managed code events (string, integer, double and guid, I'm a bit cheating with GUID as I've been processing it as string... final properly packed GUID would be much faster)
• Note that every event also gets decorated with all possible CS4.x Part A properties, like system state, device id, timestamps, etc... many others. So it's a "bulky" event, not just 4 raw fields.
• 10,000 events per second can be achieved on an oldish 2015 Macbook Pro with one thread (not a server machine)
• 50,000 events per second on more modern i7-9750H at 2.6GHz with one thread (gaming laptop, lol)
• no memory leaks (ahem).
• no memory fragmentation (ahem-ahem...).

Reason 3. We can do the rest of "incremental" work to provide a compat layer for Aria C# Server customers on top of our 1DS C++ SDK.

We can also deprecate our C++/CX and C++/CLI projection layers. Those two other technologies to generate Windows C# wrapper are "legacy" and "deprecated", we should switch to netcore P/Invoke instead.

Reason 4. I wanted to learn more about data marshaling from managed to native via P/Invoke 😄(out of curiosity ... to learn something new)

Perf numbers for a quick test to emit 100000 events in ONE thread in a loop (not multi-threaded, just 1 core, 1 background disk flush thread):

...
SDK version: 3.3.0
>>> evt_open...
handle=2933216226
>>> evt_pause...
>>> evt_log...
Elapsed    = 00:00:09.3721759
Event rate = 10669.880833115818 eps
Latency    = 0.093721759 ms
Mem used   = 6208 bytes
Fragmented = -37536 bytes
>>> evt_close...
result=0

[Max Golovanov (maxgolov)]

Usage instructions for building shared library instead of static:

bash-3.2$ ./build.sh -h
Usage: build.sh [clean] [noroot] [release] [-h|-?] [-l (static|shared)] [-D CMAKE_OPTION]

options:

 -h | -?             - this help.
 -l [static|shared]  - build static (default) or shared library.
 -D [CMAKE_OPTION]   - additional option to pass to cmake.

cmake options can be passed using CMAKE_OPTS environment variable.

To build shared library:

./build.sh -l shared

or - new way of passing options from script to cmake:

CMAKE_OPTS=-DBUILD_SHARED_LIBS=ON ./build.sh

[Max Golovanov (maxgolov)]

bemartin - I refreshed my old sample to work with latest SDK. Basically this is yet another lean and mean approach to events logging via 1DS C API. Tested on Mac-x64 and Windows-x64. But should work on Linux too, as it's pretty much common core code via C API and passing C structs around.. Of course it's missing a bunch of "semantic context" APIs (which are all just syntactic sugar around LogEvent API call), so if we ever wanted to get super-high performance .NET Core on top of 1DS, we can go that route.

I tested on my "average" laptop, and I measure about 45-50K events per second, for an average event with a few fields. This is comparable to what we had in native code a few years ago, so from overall perf perspective - very minor penalty. Even if a process logs thousands events per second, it's all relatively cheap.

It'd be fun to measure this vs. Xamarin on top of iOS/Java projections, but as we discussed - for mobile apps we do not have super-high perf requirements anyways... I was just thinking that if performance ever becomes an issue, we can incrementally build a full-fledged compatible C# API around this C API approach.

[Max Golovanov (maxgolov)]

I'm closing this stale PR due to BFG repo cleanup. Please reopen it when ready.