From f3782581e32a0b89cab7b4e99298eb27cf9ad4a1 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Mon, 21 Apr 2025 19:54:00 -0400 Subject: [PATCH 01/14] [Docs][Ipc Protocol] Fix and clarify existing spec Update CommandSets Command IDs Move EventPipe StopTracing to beginning Fix sample payload serialization Clarify Header Size and NetTrace format version Clarify that filter_data can be 0 length to avoid confusion with optional meaning that encoding can be skipped --- documentation/design-docs/ipc-protocol.md | 243 +++++++++------------- 1 file changed, 93 insertions(+), 150 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index bbff225b54..c0820e8867 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -195,92 +195,26 @@ Payloads are either encoded as fixed size structures that can be `memcpy`'ed , _ * `array` = uint length, length # of `T`s * `string` = (`array` where the last `wchar` must = `0`) or (length = `0`) -As an example, the CollectTracing command to EventPipe (explained below) encodes its Payload as: +As an example, the [CollectTracing](#collecttracing) command to EventPipe encodes its Payload as: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + - + @@ -288,27 +222,27 @@ As an example, the CollectTracing command to EventPipe (explained below) encodes - - + - + + - + - - + - + +
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677781 - 1415 - 1617 - 1819 - 2021 - 2425 - 2829 - 3233 - 4041 - 4445 - 4849 - 7677 - 80
HeaderPayloadPayload
magiccommand reserved circularBufferMBoutputPath LengthoutputPath Stringformat n Providers Keywords logLevel provider_name lengthprovider_name stringprovider_name stringfilter_data length
"DOTNET_IPC_V1"7880 0x0202 0x0000 25016"/tmp/foo.nettrace"1 1 100 2 14"MyEventSource""MyEventSource"0
@@ -359,7 +293,9 @@ See: [EventPipe Commands](#EventPipe-Commands) enum class DumpCommandId : uint8_t { // reserved = 0x00, - CreateCoreDump = 0x01, + GenerateCoreDump = 0x01, + GenerateCoreDump2 = 0x02, + GenerateCoreDump3 = 0x03, // future } ``` @@ -370,6 +306,7 @@ enum class ProfilerCommandId : uint8_t { // reserved = 0x00, AttachProfiler = 0x01, + StartupProfiler = 0x02, // future } ``` @@ -378,14 +315,15 @@ See: [Profiler Commands](#Profiler-Commands) ```c++ enum class ProcessCommandId : uint8_t { - ProcessInfo = 0x00, - ResumeRuntime = 0x01, - ProcessEnvironment = 0x02, - ProcessInfo2 = 0x04, - EnablePerfMap = 0x05, - DisablePerfMap = 0x06, - ApplyStartupHook = 0x07 - ProcessInfo3 = 0x08, + ProcessInfo = 0x00, + ResumeRuntime = 0x01, + ProcessEnvironment = 0x02, + SetEnvironmentVariable = 0x03, + ProcessInfo2 = 0x04, + EnablePerfMap = 0x05, + DisablePerfMap = 0x06, + ApplyStartupHook = 0x07 + ProcessInfo3 = 0x08, // future } ``` @@ -418,6 +356,45 @@ EventPipe Payloads are encoded with the following rules: * `array` = uint length, length # of `T`s * `string` = (`array` where the last `wchar` must = `0`) or (length = `0`) +### `StopTracing` + +Command Code: `0x0201` + +The `StopTracing` command is used to stop a specific streaming session. Clients are expected to use this command to stop streaming sessions started with [`CollectStreaming`](#CollectStreaming). + +#### Inputs: + +Header: `{ Magic; 28; 0x0201; 0x0000 }` + +Payload: +* `ulong sessionId`: The ID for the streaming session to stop + +#### Returns: + +Header: `{ Magic; 28; 0xFF00; 0x0000 }` + +Payload: +* `ulong sessionId`: the ID for the streaming session that was stopped + + +##### Details: + +Inputs: +```c +Payload +{ + ulong sessionId +} +``` + +Returns: +```c +Payload +{ + ulong sessionId +} +``` + ### `CollectTracing` Command Code: `0x0202` @@ -432,17 +409,18 @@ If the stream is stopped prematurely due to a client or server error, the `nettr #### Inputs: -Header: `{ Magic; Size; 0x0202; 0x0000 }` +Header: `{ Magic; 20 + Payload Size; 0x0202; 0x0000 }` +Payload: * `uint circularBufferMB`: The size of the circular buffer used for buffering event data while streaming -* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace format +* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace V4 format * `array providers`: The providers to turn on for the streaming session A `provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this providers * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider -* `string filter_data` (optional): Filter information +* `string filter_data`: (Callback filter information) or (length = `0`) > see ETW documentation for a more detailed explanation of Keywords, Filters, and Log Level. @@ -469,7 +447,7 @@ provider_config ulong keywords, uint logLevel, string provider_name, - string filter_data (optional) + string filter_data } ``` @@ -490,10 +468,10 @@ The `CollectTracing2` command is an extension of the `CollectTracing` command - #### Inputs: -Header: `{ Magic; Size; 0x0203; 0x0000 }` +Header: `{ Magic; 20 + Payload Size; 0x0203; 0x0000 }` * `uint circularBufferMB`: The size of the circular buffer used for buffering event data while streaming -* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace format +* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace V4 format * `bool requestRundown`: Indicates whether rundown should be fired by the runtime. * `array providers`: The providers to turn on for the streaming session @@ -501,7 +479,7 @@ A `provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this providers * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider -* `string filter_data` (optional): Filter information +* `string filter_data`: (Callback filter information) or (length = `0`) > see ETW documentation for a more detailed explanation of Keywords, Filters, and Log Level. > @@ -529,7 +507,7 @@ provider_config ulong keywords, uint logLevel, string provider_name, - string filter_data (optional) + string filter_data } ``` @@ -550,10 +528,10 @@ The `CollectTracing3` command is an extension of the `CollectTracing2` command - #### Inputs: -Header: `{ Magic; Size; 0x0203; 0x0000 }` +Header: `{ Magic; 20 + Payload Size; 0x0203; 0x0000 }` * `uint circularBufferMB`: The size of the circular buffer used for buffering event data while streaming -* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace format +* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace V4 format * `bool requestRundown`: Indicates whether rundown should be fired by the runtime. * `bool requestStackwalk`: Indicates whether stacktrace information should be recorded. * `array providers`: The providers to turn on for the streaming session @@ -562,7 +540,7 @@ A `provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this providers * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider -* `string filter_data` (optional): Filter information +* `string filter_data`: (Callback filter information) or (length = `0`) > see ETW documentation for a more detailed explanation of Keywords, Filters, and Log Level. > @@ -591,7 +569,7 @@ provider_config ulong keywords, uint logLevel, string provider_name, - string filter_data (optional) + string filter_data } ``` @@ -608,7 +586,7 @@ Followed by an Optional Continuation of a `nettrace` format stream of events. Command Code: `0x0205` -The `CollectTracing4` command is an extension of the `CollectTracing3` command - its behavior is the same as `CollectTracing3` command, except the requestRundown field is replaced by the rundownKeyword field to allow customizing the set of rundown events to be fired. +The `CollectTracing4` command is an extension of the `CollectTracing3` command - its behavior is the same as `CollectTracing3` command, except the requestRundown field is replaced by the rundownKeyword field to allow customizing the set of rundown events to be fired. A rundown keyword of `0x80020139` has the equivalent behavior as `CollectTracing3` with `requestRundown=true` and rundown keyword of `0` has the equivalent behavior as `requestRundown=false`. @@ -617,18 +595,20 @@ A rundown keyword of `0x80020139` has the equivalent behavior as `CollectTracing #### Inputs: -Header: `{ Magic; Size; 0x0205; 0x0000 }` +Header: `{ Magic; 20 + Payload Size; 0x0205; 0x0000 }` +Payload: * `uint circularBufferMB`: The size of the circular buffer used for buffering event data while streaming -* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace format +* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace V4 format * `ulong rundownKeyword`: Indicates the keyword for the rundown provider +* `bool requestStackwalk`: Indicates whether stacktrace information should be recorded. * `array providers`: The providers to turn on for the streaming session A `provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this providers * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider -* `string filter_data` (optional): Filter information +* `string filter_data`: (Callback filter information) or (length = `0`) > see ETW documentation for a more detailed explanation of Keywords, Filters, and Log Level. > @@ -656,7 +636,7 @@ provider_config ulong keywords, uint logLevel, string provider_name, - string filter_data (optional) + string filter_data } ``` @@ -669,43 +649,6 @@ Payload ``` Followed by an Optional Continuation of a `nettrace` format stream of events. -### `StopTracing` - -Command Code: `0x0201` - -The `StopTracing` command is used to stop a specific streaming session. Clients are expected to use this command to stop streaming sessions started with [`CollectStreaming`](#CollectStreaming). - -#### Inputs: - -Header: `{ Magic; 28; 0x0201; 0x0000 }` - -* `ulong sessionId`: The ID for the streaming session to stop - -#### Returns: - -Header: `{ Magic; 28; 0xFF00; 0x0000 }` - -* `ulong sessionId`: the ID for the streaming session that was stopped - - -##### Details: - -Inputs: -```c -Payload -{ - ulong sessionId -} -``` - -Returns: -```c -Payload -{ - ulong sessionId -} -``` - ## Dump Commands ### `CreateCoreDump` From 8b5bf4606911233667a4638ede63c862d61a9647 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Mon, 21 Apr 2025 19:54:26 -0400 Subject: [PATCH 02/14] [Docs][IPC Protocol] Add CollectTracing5 --- documentation/design-docs/ipc-protocol.md | 397 +++++++++++++++++++++- 1 file changed, 394 insertions(+), 3 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index c0820e8867..9ece92e147 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -285,6 +285,7 @@ enum class EventPipeCommandId : uint8_t CollectTracing2 = 0x03, // create/start a given session with/without rundown CollectTracing3 = 0x04, // create/start a given session with/without collecting stacks CollectTracing4 = 0x05, // create/start a given session with specific rundown keyword + CollectTracing5 = 0x06, // create/start a given session with/without user_events } ``` See: [EventPipe Commands](#EventPipe-Commands) @@ -335,6 +336,59 @@ For example, the Command to start a stream session with EventPipe would be `0x02 ## EventPipe Commands +The EventPipe CommandSet enables Clients to create/start or stop [EventPipe](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/eventpipe) Sessions. Two session types are currently supported through this protocol. + +### Streaming Session + +The latest CommandID [`CollectTracing5`](#collecttracing5) supports configuring the following streaming EventPipe session options: +* `uint circularBufferMB`: The size of the circular buffer used for buffering event data +* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace V4 format +* `ulong rundownKeyword`: Indicates the keyword for the rundown provider +* `bool requestStackwalk`: Indicates whether stacktrace information should be recorded. +* `array providers`: The providers to turn on for the session + +### User_events Session + +The latest CommandID [`CollectTracing5`](#collecttracing5) supports configuring the following user_events EventPipe session options: + +Payload: +* `ulong rundownKeyword`: Indicates the keyword for the rundown provider +* `array providers`: The providers to turn on for the session + +### Session Providers + +The `provider_config` is composed of the following data: +* `ulong keywords`: The keywords to turn on with this provider +* `uint logLevel`: The level of information to turn on +* `string provider_name`: The name of the provider +* `string filter_data`: (Callback filter information) or (length = `0`) +* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). See [details](#event-filter). + +For user_events EventPipe Session Providers, another `provider_config` field is configurable: +* `tracepoint_config config`: Maps Event IDs to tracepoints. If an Event ID is excluded by `event_filter`, it will not be written to any tracepoint. See [details](#tracepoint-config) + +#### Event Filter +An `event_filter` is comprised of the following data: +* `bool allow`: 0 for deny list, 1 for allow list +* `array event_ids`: List of Event IDs to deny or allow. + +See [event_filter serialization examples](#event_filter) + +#### Tracepoint Config +A `tracepoint_config` is comprised of the following data: +* `string default_tracepoint_name`: (The default tracepoint filtered Event IDs will be written to unless otherwise specified by `tracepoints`) or (length = `0` to only write to tracepoints specified in `tracepoints`) +* `array tracepoints`: Specifies alternate tracepoints for a set of Event IDs to be written to instead of the default tracepoint or (length = `0`). + +A `tracepoint_set` is comprised of the following data: +* `string tracepoint_name`: The tracepoint that the following subset of Event IDs should be written to. +* `array event_ids`: The Event IDs to be written to `tracepoint_name`. + +With a user_events session, atleast one of `default_tracepoint_name` and `tracepoints` must be specified. An error will be returned through the stream if both are length = `0`. +Event IDs specified in `tracepoint_set`s must be exclusive. If an Event ID is detected in different `tracepoint_set`s of the provider, an error will be returned through the stream. + +See [tracepoint_config serialization examples](#tracepoint_config) + +### EventPipe Command IDs ```c++ enum class EventPipeCommandId : uint8_t { @@ -344,6 +398,7 @@ enum class EventPipeCommandId : uint8_t CollectTracing2 = 0x03, // create/start a given session with/without rundown CollectTracing3 = 0x04, // create/start a given session with/without collecting stacks CollectTracing4 = 0x05, // create/start a given session with specific rundown keyword + CollectTracing5 = 0x06, // create/start a given session with/without user_events } ``` EventPipe Payloads are encoded with the following rules: @@ -360,21 +415,21 @@ EventPipe Payloads are encoded with the following rules: Command Code: `0x0201` -The `StopTracing` command is used to stop a specific streaming session. Clients are expected to use this command to stop streaming sessions started with [`CollectStreaming`](#CollectStreaming). +The `StopTracing` command is used to stop a specific EventPipe session. Clients are expected to use this command to stop EventPipe sessions started with [`CollectStreaming`](#CollectStreaming). #### Inputs: Header: `{ Magic; 28; 0x0201; 0x0000 }` Payload: -* `ulong sessionId`: The ID for the streaming session to stop +* `ulong sessionId`: The ID for the EventPipe session to stop #### Returns: Header: `{ Magic; 28; 0xFF00; 0x0000 }` Payload: -* `ulong sessionId`: the ID for the streaming session that was stopped +* `ulong sessionId`: the ID for the EventPipe session that was stopped ##### Details: @@ -649,6 +704,342 @@ Payload ``` Followed by an Optional Continuation of a `nettrace` format stream of events. +### `CollectTracing5` + +Command Code: `0x0206` + +The `CollectTracing5` command is an extension of the `CollectTracing4` command. It has all the capabilities of `CollectTracing4` and introduces new fields to enable a Linux-only user_events-based eventpipe session and to prescribe an allow/deny list for Event IDs. When the user_events-based eventpipe session is enabled, the file descriptor and SCM_RIGHTS of the `user_events_data` file must be sent through the optional continuation stream as [described](#passing_file_descriptor). The runtime will register tracepoints based on the provider configurations passed in, and runtime events will be written directly to the `user_events_data` file descriptor. The allow/deny list of Event IDs will apply after the keyword/level filter to determine whether or not that provider's event will be written. When using this command, even without leveraging the new user_events-based eventpipe session option, the new fields must be serialized. + +> Note available for .NET 10.0 and later. + +#### Inputs: + +Header: `{ Magic; 20 + Payload Size; 0x0206; 0x0000 }` + +#### [Streaming Session](#streaming-session) Payload: +* `uint output_format`: 0 +* `uint circularBufferMB`: The size of the circular buffer used for buffering event data +* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace V4 format +* `ulong rundownKeyword`: Indicates the keyword for the rundown provider +* `bool requestStackwalk`: Indicates whether stacktrace information should be recorded. +* `array providers`: The providers to turn on for the session + +The Streaming Session `provider_config` is composed of the following data: +* `ulong keywords`: The keywords to turn on with this provider +* `uint logLevel`: The level of information to turn on +* `string provider_name`: The name of the provider +* `string filter_data`: (Callback filter information) or (length = `0`) +* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). See [details](#event-filter). + +#### [User_events Session](#user_events-session) Payload: +* `uint output_format`: 1 +* `ulong rundownKeyword`: Indicates the keyword for the rundown provider +* `array providers`: The providers to turn on for the session + +The User_events Session `provider_config` is composed of the following data: +* `ulong keywords`: The keywords to turn on with this provider +* `uint logLevel`: The level of information to turn on +* `string provider_name`: The name of the provider +* `string filter_data`: (Callback filter information) or (length = `0`) +* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). See [details](#event-filter). +* `tracepoint_config config`: Maps Event IDs to tracepoints. If an Event ID is excluded by `event_filter`, it will not be written to any tracepoint. See [details](#tracepoint-config) + +> See ETW documentation for a more detailed explanation of Keywords, Filters, and Log Level. + +#### Returns (as an IPC Message Payload): + +Header: `{ Magic; 28; 0xFF00; 0x0000; }` + +`CollectTracing5` returns: +* `ulong sessionId`: the ID for the EventPipe Session started + +A Streaming Session started with `CollectTracing5` is followed by an Optional Continuation of a `nettrace` format stream of events. + +A User_events Session started with `CollectTracing5` expects the Optional Continuation to contain another message passing along the SCM_RIGHTS `user_events_data` file descriptor. See [details](#passing_file_descriptor) + +## EventPipe Payload Serialization Examples + +### Event_filter +Example `event_filter` serialization. Serializing +``` +allow=0, event_ids=[]: Allow all events. +``` + + + + + + + + + + + + + + + + + + + +
12-5
boolarray<uint>
allowevent_ids
00
+ +``` +allow=0, event_ids=[4, 5]: Deny only Event IDs 4 and 5. +``` + + + + + + + + + + + + + + + + + + + + + + +
12610-13
boolarray<uint>
allowevent_ids
0245
+ +``` +allow=1, event_ids=[]: Deny all events. +``` + + + + + + + + + + + + + + + + + + + + +
12-5
boolarray<uint>
allowevent_ids
10
+ +``` +allow=1, event_ids=[1, 2, 3]: Allow only Event IDs 1, 2, and 3. +``` + + + + + + + + + + + + + + + + + + + + + + + + + + +
1261014-17
boolarray<uint>
allowevent_ids
13123
+ +### Tracepoint_config +Example `tracepoint_config` serialization +``` +Output_format=0, DO NOT encode bytes for tracepoint_config +Output_format=1, encode bytes for tracepoint_config +``` + +``` +All allowed Event IDs will be written to a default "MyTracepoint" tracepoint +``` + + + + + + + + + + + + + + + + + + + + +
1533-36
string (array<wchar>)array<uint>
default_tracepoint_nametracepoints
14"MyTracepoint"0
+ +``` +Allowed Event IDs 1 - 9 will be written to tracepoint "LowEvents". +All other allowed Event IDs will be written to "MyTracepoint" +``` + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
1533374161656973778185899397-100
string (array<wchar>)uintstring (array<wchar>)array<uint>
default_tracepoint_nametracepointstracepoint_nameevent_ids
14"MyTracepoint"110"LowEvents"9123456789
+ +``` +Allowed Event IDs 1 - 9 will be written to tracepoint "LowEvents". +No default tracepoint needed, don't write any other allowed Event IDs +``` + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
1591333374145495357616569-72
string (array<wchar>)uintstring (array<wchar>)array<uint>
default_tracepoint_nametracepointstracepoint_nameevent_ids
0110"LowEvents"9123456789
+ +### passing_file_descriptor + +> Note: This only applies to enabling an user_event-based EventPipe session, which is specifically a Linux feature + +To register [user_event](https://docs.kernel.org/trace/user_events.html) tracepoints and write events, access to the root protected `user_events_data` file is required. Once the .NET Runtime's Diagnostic Server processes a [CollectTracing5](#collecttracing5) command specifying the `user_events` format (`output_format=1`), it expects that the client will send a file descriptor to the [continuation stream](#general-flow) via SCM_RIGHTS. + +```C +#include +#include + +struct msghdr { + void * msg_name; /* ignored by runtime */ + unsigned int msg_namelen; /* ignored by runtime */ + struct iovec * msg_iov; /* runtime will "parse" 1 byte */ + unsigned int msg_iovlen; /* runtime will "parse" one msg_iov */ + void * msg_control; /* ancillary data */ + unsigned int msg_controllen; /* ancillary data buffer len */ + int msg_flags; /* ignored by runtime */ +}; + +struct cmsghdr { + unsigned int cmsg_len; /* length of control message */ + int cmsg_level; /* SOL_SOCKET */ + int cmsg_type; /* SCM_RIGHTS */ + int cmsg_data[0]; /* file descriptor */ +}; +``` + +For parsing the file descriptor passed with SCM_RIGHTS, the runtime will `recvmsg` the message and only care about the control message containing ancillary data. It will read one byte from the `msg_iov` buffer just to receive the ancillary data, but it will disregard the contents of the `msg_iov` buffers. + ## Dump Commands ### `CreateCoreDump` From efe387a9b55f7a57e646edca7ab505862085209b Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Mon, 28 Apr 2025 11:05:25 -0400 Subject: [PATCH 03/14] Confine CollectTracing5 details to its section --- documentation/design-docs/ipc-protocol.md | 99 +++++++++-------------- 1 file changed, 36 insertions(+), 63 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index 9ece92e147..4dc38ad5c8 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -336,58 +336,6 @@ For example, the Command to start a stream session with EventPipe would be `0x02 ## EventPipe Commands -The EventPipe CommandSet enables Clients to create/start or stop [EventPipe](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/eventpipe) Sessions. Two session types are currently supported through this protocol. - -### Streaming Session - -The latest CommandID [`CollectTracing5`](#collecttracing5) supports configuring the following streaming EventPipe session options: -* `uint circularBufferMB`: The size of the circular buffer used for buffering event data -* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace V4 format -* `ulong rundownKeyword`: Indicates the keyword for the rundown provider -* `bool requestStackwalk`: Indicates whether stacktrace information should be recorded. -* `array providers`: The providers to turn on for the session - -### User_events Session - -The latest CommandID [`CollectTracing5`](#collecttracing5) supports configuring the following user_events EventPipe session options: - -Payload: -* `ulong rundownKeyword`: Indicates the keyword for the rundown provider -* `array providers`: The providers to turn on for the session - -### Session Providers - -The `provider_config` is composed of the following data: -* `ulong keywords`: The keywords to turn on with this provider -* `uint logLevel`: The level of information to turn on -* `string provider_name`: The name of the provider -* `string filter_data`: (Callback filter information) or (length = `0`) -* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). See [details](#event-filter). - -For user_events EventPipe Session Providers, another `provider_config` field is configurable: -* `tracepoint_config config`: Maps Event IDs to tracepoints. If an Event ID is excluded by `event_filter`, it will not be written to any tracepoint. See [details](#tracepoint-config) - -#### Event Filter -An `event_filter` is comprised of the following data: -* `bool allow`: 0 for deny list, 1 for allow list -* `array event_ids`: List of Event IDs to deny or allow. - -See [event_filter serialization examples](#event_filter) - -#### Tracepoint Config -A `tracepoint_config` is comprised of the following data: -* `string default_tracepoint_name`: (The default tracepoint filtered Event IDs will be written to unless otherwise specified by `tracepoints`) or (length = `0` to only write to tracepoints specified in `tracepoints`) -* `array tracepoints`: Specifies alternate tracepoints for a set of Event IDs to be written to instead of the default tracepoint or (length = `0`). - -A `tracepoint_set` is comprised of the following data: -* `string tracepoint_name`: The tracepoint that the following subset of Event IDs should be written to. -* `array event_ids`: The Event IDs to be written to `tracepoint_name`. - -With a user_events session, atleast one of `default_tracepoint_name` and `tracepoints` must be specified. An error will be returned through the stream if both are length = `0`. -Event IDs specified in `tracepoint_set`s must be exclusive. If an Event ID is detected in different `tracepoint_set`s of the provider, an error will be returned through the stream. - -See [tracepoint_config serialization examples](#tracepoint_config) - ### EventPipe Command IDs ```c++ enum class EventPipeCommandId : uint8_t @@ -708,7 +656,7 @@ Followed by an Optional Continuation of a `nettrace` format stream of events. Command Code: `0x0206` -The `CollectTracing5` command is an extension of the `CollectTracing4` command. It has all the capabilities of `CollectTracing4` and introduces new fields to enable a Linux-only user_events-based eventpipe session and to prescribe an allow/deny list for Event IDs. When the user_events-based eventpipe session is enabled, the file descriptor and SCM_RIGHTS of the `user_events_data` file must be sent through the optional continuation stream as [described](#passing_file_descriptor). The runtime will register tracepoints based on the provider configurations passed in, and runtime events will be written directly to the `user_events_data` file descriptor. The allow/deny list of Event IDs will apply after the keyword/level filter to determine whether or not that provider's event will be written. When using this command, even without leveraging the new user_events-based eventpipe session option, the new fields must be serialized. +The `CollectTracing5` command is an extension of the `CollectTracing4` command. It has all the capabilities of `CollectTracing4` and introduces new fields to enable a Linux-only user_events-based eventpipe session and to prescribe an allow/deny list for Event IDs. When the user_events-based eventpipe session is enabled, the file descriptor and SCM_RIGHTS of the `user_events_data` file must be sent through the optional continuation stream as [described](#passing_file_descriptor). The runtime will register tracepoints based on the provider configurations passed in, and runtime events will be written directly to the `user_events_data` file descriptor. The allow/deny list of Event IDs will apply after the keyword/level filter to determine whether or not that provider's event will be written. > Note available for .NET 10.0 and later. @@ -716,33 +664,58 @@ The `CollectTracing5` command is an extension of the `CollectTracing4` command. Header: `{ Magic; 20 + Payload Size; 0x0206; 0x0000 }` -#### [Streaming Session](#streaming-session) Payload: +#### Streaming Session Payload: * `uint output_format`: 0 * `uint circularBufferMB`: The size of the circular buffer used for buffering event data * `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace V4 format * `ulong rundownKeyword`: Indicates the keyword for the rundown provider * `bool requestStackwalk`: Indicates whether stacktrace information should be recorded. -* `array providers`: The providers to turn on for the session +* `array providers`: The providers to turn on for the session -The Streaming Session `provider_config` is composed of the following data: +The `streaming_provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this provider * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider * `string filter_data`: (Callback filter information) or (length = `0`) -* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). See [details](#event-filter). +* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). + +An `event_filter` is comprised of the following data: +* `bool allow`: 0 for deny list, 1 for allow list +* `array event_ids`: List of Event IDs to deny or allow. + +See [event_filter serialization examples](#event_filter) -#### [User_events Session](#user_events-session) Payload: +#### User_events Session Payload: * `uint output_format`: 1 * `ulong rundownKeyword`: Indicates the keyword for the rundown provider -* `array providers`: The providers to turn on for the session +* `array providers`: The providers to turn on for the session -The User_events Session `provider_config` is composed of the following data: +The `user_events_provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this provider * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider * `string filter_data`: (Callback filter information) or (length = `0`) -* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). See [details](#event-filter). -* `tracepoint_config config`: Maps Event IDs to tracepoints. If an Event ID is excluded by `event_filter`, it will not be written to any tracepoint. See [details](#tracepoint-config) +* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). +* `tracepoint_config config`: Maps Event IDs to tracepoints. If an Event ID is excluded by `event_filter`, it will not be written to any tracepoint. + +An `event_filter` is comprised of the following data: +* `bool allow`: 0 for deny list, 1 for allow list +* `array event_ids`: List of Event IDs to deny or allow. + +See [event_filter serialization examples](#event_filter) + +A `tracepoint_config` is comprised of the following data: +* `string default_tracepoint_name`: (The default tracepoint filtered Event IDs will be written to unless otherwise specified by `tracepoints`) or (length = `0` to only write to tracepoints specified in `tracepoints`) +* `array tracepoints`: Specifies alternate tracepoints for a set of Event IDs to be written to instead of the default tracepoint or (length = `0`). + +A `tracepoint_set` is comprised of the following data: +* `string tracepoint_name`: The tracepoint that the following subset of Event IDs should be written to. +* `array event_ids`: The Event IDs to be written to `tracepoint_name`. + +With a user_events session, atleast one of `default_tracepoint_name` and `tracepoints` must be specified. An error will be returned through the stream if both are length = `0`. +Event IDs specified in `tracepoint_set`s must be exclusive. If an Event ID is detected in different `tracepoint_set`s of the provider, an error will be returned through the stream. + +See [tracepoint_config serialization examples](#tracepoint_config) > See ETW documentation for a more detailed explanation of Keywords, Filters, and Log Level. @@ -871,7 +844,7 @@ allow=1, event_ids=[1, 2, 3]: Allow only Event IDs 1, 2, and 3. ### Tracepoint_config Example `tracepoint_config` serialization ``` -Output_format=0, DO NOT encode bytes for tracepoint_config +Output_format=0, Streaming Sessions DO NOT encode bytes for tracepoint_config Output_format=1, encode bytes for tracepoint_config ``` From 87902f050ea952867e815f33cf206a8a36bbaba1 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Wed, 30 Apr 2025 15:08:58 -0400 Subject: [PATCH 04/14] [Docs][IPC Protocol] Detail user_events format --- documentation/design-docs/ipc-protocol.md | 33 +++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index 4dc38ad5c8..8997751e2f 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -1013,6 +1013,39 @@ struct cmsghdr { For parsing the file descriptor passed with SCM_RIGHTS, the runtime will `recvmsg` the message and only care about the control message containing ancillary data. It will read one byte from the `msg_iov` buffer just to receive the ancillary data, but it will disregard the contents of the `msg_iov` buffers. +### User_events format + +When writing events to their mapped user_events tracepoints prescribed by the `tracepoint_config` in the [User_events session payload](#user_events-session-payload), the runtime will adapt the [user_events writing protocol](https://docs.kernel.org/trace/user_events.html#writing-data) to write the event as: + +``` +struct iovec io[5]; + +io[0].iov_base = &myTracepointIndex; // __u32 from event_reg +io[0].iov_len = sizeof(myTracepointIndex); +io[1].iov_base = &event_id // EventID defined by EventSource/native manifest +io[1].iov_len = sizeof(event_id) +io[2].iov_base = &this_event_payload; // __rel_loc char[] +io[2].iov_len = sizeof(this_event_payload); +io[3].iov_base = &this_event_meta; // __rel_loc char[] +io[3].iov_len = sizeof(this_event_meta); +io[4].iov_base = &actual_data; // char[] +io[4].iov_len = actual_data_len; + +writev(ep_session->data_fd, (const struct iovec *)io, 5); +``` + +The `__rel_loc` is the relative dynamic array attribute described [here](https://lwn.net/Articles/876682/). + +The payload points at a blob of data with the same format as an EventPipe payload – the concatenated encoded values for all the parameters + +The metadata either points at nothing if the event doesn’t have metadata, or it points at a metadata blob matching the NetTrace version 5 formatting convention. Specifically it is the data that would be stored inside the PayloadBytes area of an event blob within a MetadataBlock described [here](https://github.com/microsoft/perfview/blob/main/src/TraceEvent/EventPipe/NetTraceFormat_v5.md#metadata-event-encoding). + +> NOTE: V5 and V6 metadata formats have the same info, but they aren’t formatted identically. Parsing and reserialization is required to convert between the two. + +### Which events have metadata? + +The runtime will keep track per-session whether it has sent a particular event before. The first time each event is sent during a session, metadata will be included, and otherwise, it will be left empty. As a special case, runtime events currently implemented in native code will never send metadata. + ## Dump Commands ### `CreateCoreDump` From 870e337729486ab7b50cf94341f4f5fa3fdba3f2 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Mon, 5 May 2025 14:46:41 -0400 Subject: [PATCH 05/14] Add User_events format string --- documentation/design-docs/ipc-protocol.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index 8997751e2f..07ad013c57 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -1015,13 +1015,23 @@ For parsing the file descriptor passed with SCM_RIGHTS, the runtime will `recvms ### User_events format +#### User_events Registration + +Once the runtime has received the configured tracepoint names as detailed under [tracepoint_config](#user_events-session-payload), it uses the [file descriptor passed in the continuation stream](#passing_file_descriptor) to register the prescribed tracepoint names following the [user_events registering protocol](https://docs.kernel.org/trace/user_events.html#registering). The runtime will construct a `user_reg` struct for every tracepoint name, defaulting to using none of the `user_reg` flags, so the resulting command format will be as follows: + +` u16 event_id; __rel_loc char[] payload; __rel_loc char[] meta` + +See [user_events writing](#user_events-writing) below for field details`. + +#### User_events Writing + When writing events to their mapped user_events tracepoints prescribed by the `tracepoint_config` in the [User_events session payload](#user_events-session-payload), the runtime will adapt the [user_events writing protocol](https://docs.kernel.org/trace/user_events.html#writing-data) to write the event as: ``` struct iovec io[5]; -io[0].iov_base = &myTracepointIndex; // __u32 from event_reg -io[0].iov_len = sizeof(myTracepointIndex); +io[0].iov_base = &my_tracepoint_index; // __u32 from event_reg +io[0].iov_len = sizeof(my_tracepoint_index); io[1].iov_base = &event_id // EventID defined by EventSource/native manifest io[1].iov_len = sizeof(event_id) io[2].iov_base = &this_event_payload; // __rel_loc char[] From 55f287708596666e330a4fe70092cb9d2428b185 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Wed, 7 May 2025 11:36:44 -0400 Subject: [PATCH 06/14] Encode user_events payload as u8 --- documentation/design-docs/ipc-protocol.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index 07ad013c57..7fcc1f197c 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -1019,7 +1019,7 @@ For parsing the file descriptor passed with SCM_RIGHTS, the runtime will `recvms Once the runtime has received the configured tracepoint names as detailed under [tracepoint_config](#user_events-session-payload), it uses the [file descriptor passed in the continuation stream](#passing_file_descriptor) to register the prescribed tracepoint names following the [user_events registering protocol](https://docs.kernel.org/trace/user_events.html#registering). The runtime will construct a `user_reg` struct for every tracepoint name, defaulting to using none of the `user_reg` flags, so the resulting command format will be as follows: -` u16 event_id; __rel_loc char[] payload; __rel_loc char[] meta` +` u16 event_id; __rel_loc u8[] payload; __rel_loc u8[] meta` See [user_events writing](#user_events-writing) below for field details`. @@ -1034,11 +1034,11 @@ io[0].iov_base = &my_tracepoint_index; // __u32 from event_reg io[0].iov_len = sizeof(my_tracepoint_index); io[1].iov_base = &event_id // EventID defined by EventSource/native manifest io[1].iov_len = sizeof(event_id) -io[2].iov_base = &this_event_payload; // __rel_loc char[] +io[2].iov_base = &this_event_payload; // __rel_loc u8[] io[2].iov_len = sizeof(this_event_payload); -io[3].iov_base = &this_event_meta; // __rel_loc char[] +io[3].iov_base = &this_event_meta; // __rel_loc u8[] io[3].iov_len = sizeof(this_event_meta); -io[4].iov_base = &actual_data; // char[] +io[4].iov_base = &actual_data; // u8[] io[4].iov_len = actual_data_len; writev(ep_session->data_fd, (const struct iovec *)io, 5); From c2cb3db17637621a7a899cb69887998f1e9f0251 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Mon, 12 May 2025 13:42:07 -0400 Subject: [PATCH 07/14] Update tracepoint format --- documentation/design-docs/ipc-protocol.md | 44 +++++++++++++++-------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index 7fcc1f197c..27b615726f 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -1019,7 +1019,9 @@ For parsing the file descriptor passed with SCM_RIGHTS, the runtime will `recvms Once the runtime has received the configured tracepoint names as detailed under [tracepoint_config](#user_events-session-payload), it uses the [file descriptor passed in the continuation stream](#passing_file_descriptor) to register the prescribed tracepoint names following the [user_events registering protocol](https://docs.kernel.org/trace/user_events.html#registering). The runtime will construct a `user_reg` struct for every tracepoint name, defaulting to using none of the `user_reg` flags, so the resulting command format will be as follows: -` u16 event_id; __rel_loc u8[] payload; __rel_loc u8[] meta` +##### Tracepoint Format V1 + +` u8 version; u16 event_id; __rel_loc u8[] extension; __rel_loc u8[] payload; __rel_loc u8[] meta` See [user_events writing](#user_events-writing) below for field details`. @@ -1028,27 +1030,39 @@ See [user_events writing](#user_events-writing) below for field details`. When writing events to their mapped user_events tracepoints prescribed by the `tracepoint_config` in the [User_events session payload](#user_events-session-payload), the runtime will adapt the [user_events writing protocol](https://docs.kernel.org/trace/user_events.html#writing-data) to write the event as: ``` -struct iovec io[5]; +struct iovec io[7]; -io[0].iov_base = &my_tracepoint_index; // __u32 from event_reg -io[0].iov_len = sizeof(my_tracepoint_index); -io[1].iov_base = &event_id // EventID defined by EventSource/native manifest -io[1].iov_len = sizeof(event_id) -io[2].iov_base = &this_event_payload; // __rel_loc u8[] -io[2].iov_len = sizeof(this_event_payload); -io[3].iov_base = &this_event_meta; // __rel_loc u8[] -io[3].iov_len = sizeof(this_event_meta); -io[4].iov_base = &actual_data; // u8[] -io[4].iov_len = actual_data_len; +io[0].iov_base = &write_index; // __u32 tracepoint write index from registration +io[0].iov_len = sizeof(write_index); +io[1].iov_base = &version; // __u8 tracepoint format version +io[1].iov_len = sizeof(version); +io[2].iov_base = &event_id; // __u16 EventID defined by EventSource/native manifest +io[2].iov_len = sizeof(event_id); +io[3].iov_base = &extension; // __rel_loc u8[] NetTrace V6 label list +io[3].iov_base = sizeof(extension); +io[4].iov_base = &payload; // __rel_loc u8[] event payload +io[4].iov_len = sizeof(payload); +io[5].iov_base = &meta; // __rel_loc u8[] event metadata +io[5].iov_len = sizeof(meta); +io[6].iov_base = &data; // __u8[] data +io[6].iov_len = data_len; -writev(ep_session->data_fd, (const struct iovec *)io, 5); +writev(ep_session->data_fd, (const struct iovec *)io, 7); ``` The `__rel_loc` is the relative dynamic array attribute described [here](https://lwn.net/Articles/876682/). -The payload points at a blob of data with the same format as an EventPipe payload – the concatenated encoded values for all the parameters +The `write_index` is the tracepoint's write index determined during tracepoint registration. + +The `version` is the version of the tracepoint format, which in this case is [version 1](#tracepoint-format-v1). + +The `event_id` is the ID of the event, defined by the EventSource/native manifest. + +The `extension` points at a [NetTrace V6 LabelList block](https://github.com/microsoft/perfview/blob/main/src/TraceEvent/EventPipe/NetTraceFormat.md#labellistblock) describing other fields associated with the event. e.g. ActivityId, RelatedActivityId, event_thread, and stack. + +The `payload` points at a blob of data with the same format as an EventPipe payload – the concatenated encoded values for all the parameters. -The metadata either points at nothing if the event doesn’t have metadata, or it points at a metadata blob matching the NetTrace version 5 formatting convention. Specifically it is the data that would be stored inside the PayloadBytes area of an event blob within a MetadataBlock described [here](https://github.com/microsoft/perfview/blob/main/src/TraceEvent/EventPipe/NetTraceFormat_v5.md#metadata-event-encoding). +The `metadata` either points at nothing if the event doesn’t have metadata, or it points at a metadata blob matching the NetTrace version 5 formatting convention. Specifically it is the data that would be stored inside the PayloadBytes area of an event blob within a MetadataBlock described [here](https://github.com/microsoft/perfview/blob/main/src/TraceEvent/EventPipe/NetTraceFormat_v5.md#metadata-event-encoding). > NOTE: V5 and V6 metadata formats have the same info, but they aren’t formatted identically. Parsing and reserialization is required to convert between the two. From 2b9bce0c26e064bbe6ee9ea3e6cc998b071770e5 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Mon, 12 May 2025 14:45:16 -0400 Subject: [PATCH 08/14] Update event_filter example and Add bool encoding --- documentation/design-docs/ipc-protocol.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index 27b615726f..6b96aa7c12 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -356,6 +356,7 @@ EventPipe Payloads are encoded with the following rules: * `ulong` = 8 little endian bytes * `wchar` = 2 little endian bytes, UTF16 encoding * `byte` = 1 unsigned little endian byte +* `bool` = 1 unsigned little endian byte * `array` = uint length, length # of `T`s * `string` = (`array` where the last `wchar` must = `0`) or (length = `0`) @@ -735,7 +736,8 @@ A User_events Session started with `CollectTracing5` expects the Optional Contin ### Event_filter Example `event_filter` serialization. Serializing ``` -allow=0, event_ids=[]: Allow all events. +allow=0, event_ids=[] +Deny Nothing === Allow all events. ``` @@ -759,7 +761,8 @@ allow=0, event_ids=[]: Allow all events.
``` -allow=0, event_ids=[4, 5]: Deny only Event IDs 4 and 5. +allow=0, event_ids=[4, 5] +Deny only Event IDs 4 and 5 === Allow all Event IDs except 4 and 5 ``` @@ -786,7 +789,8 @@ allow=0, event_ids=[4, 5]: Deny only Event IDs 4 and 5.
``` -allow=1, event_ids=[]: Deny all events. +allow=1, event_ids=[] +Allow Nothing === Deny all events. ``` @@ -811,7 +815,8 @@ allow=1, event_ids=[]: Deny all events.
``` -allow=1, event_ids=[1, 2, 3]: Allow only Event IDs 1, 2, and 3. +allow=1, event_ids=[1, 2, 3] +Allow only EventIDs 1, 2, and 3 === Deny all EventIDs except 1, 2, and 3. ``` From d5d2222a7d70496074c894bd39dd1f3a2f04f5c3 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Tue, 27 May 2025 12:28:24 -0400 Subject: [PATCH 09/14] Clarify streaming specific fields --- documentation/design-docs/ipc-protocol.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index 6b96aa7c12..e443a043eb 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -667,8 +667,8 @@ Header: `{ Magic; 20 + Payload Size; 0x0206; 0x0000 }` #### Streaming Session Payload: * `uint output_format`: 0 -* `uint circularBufferMB`: The size of the circular buffer used for buffering event data -* `uint format`: 0 for the legacy NetPerf format and 1 for the NetTrace V4 format +* `uint streaming_circularBufferMB`: Specifies the size of the Streaming session's circular buffer used for buffering event data. +* `uint streaming_format`: 0 for the legacy NetPerf format and 1 for the NetTrace V4 format. Specifies the format in which event data will be serialized into the IPC Stream * `ulong rundownKeyword`: Indicates the keyword for the rundown provider * `bool requestStackwalk`: Indicates whether stacktrace information should be recorded. * `array providers`: The providers to turn on for the session From c9e8f966dcd6bd121c3fd1aa79d47c8fb6ee53a9 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Tue, 27 May 2025 17:04:56 -0400 Subject: [PATCH 10/14] Rename and clarify field --- documentation/design-docs/ipc-protocol.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index e443a043eb..e3c2d7c373 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -228,7 +228,7 @@ As an example, the [CollectTracing](#collecttracing) command to EventPipe encode - + @@ -424,7 +424,7 @@ A `provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this providers * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider -* `string filter_data`: (Callback filter information) or (length = `0`) +* `string arguments`: (Key-value pairs string to pass to the provider) or (length = 0) > see ETW documentation for a more detailed explanation of Keywords, Filters, and Log Level. @@ -451,7 +451,7 @@ provider_config ulong keywords, uint logLevel, string provider_name, - string filter_data + string arguments } ``` @@ -483,7 +483,7 @@ A `provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this providers * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider -* `string filter_data`: (Callback filter information) or (length = `0`) +* `string arguments`: (Key-value pairs string to pass to the provider) or (length = 0) > see ETW documentation for a more detailed explanation of Keywords, Filters, and Log Level. > @@ -511,7 +511,7 @@ provider_config ulong keywords, uint logLevel, string provider_name, - string filter_data + string arguments } ``` @@ -544,7 +544,7 @@ A `provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this providers * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider -* `string filter_data`: (Callback filter information) or (length = `0`) +* `string arguments`: (Key-value pairs string to pass to the provider) or (length = 0) > see ETW documentation for a more detailed explanation of Keywords, Filters, and Log Level. > @@ -573,7 +573,7 @@ provider_config ulong keywords, uint logLevel, string provider_name, - string filter_data + string arguments } ``` @@ -612,7 +612,7 @@ A `provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this providers * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider -* `string filter_data`: (Callback filter information) or (length = `0`) +* `string arguments`: (Key-value pairs string to pass to the provider) or (length = 0) > see ETW documentation for a more detailed explanation of Keywords, Filters, and Log Level. > @@ -640,7 +640,7 @@ provider_config ulong keywords, uint logLevel, string provider_name, - string filter_data + string arguments } ``` @@ -677,7 +677,7 @@ The `streaming_provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this provider * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider -* `string filter_data`: (Callback filter information) or (length = `0`) +* `string arguments`: (Key-value pairs string to pass to the provider) or (length = 0) * `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). An `event_filter` is comprised of the following data: @@ -695,7 +695,7 @@ The `user_events_provider_config` is composed of the following data: * `ulong keywords`: The keywords to turn on with this provider * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider -* `string filter_data`: (Callback filter information) or (length = `0`) +* `string arguments`: (Key-value pairs string to pass to the provider) or (length = 0) * `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). * `tracepoint_config config`: Maps Event IDs to tracepoints. If an Event ID is excluded by `event_filter`, it will not be written to any tracepoint. From 3f310dea92727b8c50e77159107cf0f5383bb445 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Tue, 27 May 2025 17:09:39 -0400 Subject: [PATCH 11/14] Rename for consistency --- documentation/design-docs/ipc-protocol.md | 48 +++++++++++------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index e3c2d7c373..8e005bfc7e 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -657,7 +657,7 @@ Followed by an Optional Continuation of a `nettrace` format stream of events. Command Code: `0x0206` -The `CollectTracing5` command is an extension of the `CollectTracing4` command. It has all the capabilities of `CollectTracing4` and introduces new fields to enable a Linux-only user_events-based eventpipe session and to prescribe an allow/deny list for Event IDs. When the user_events-based eventpipe session is enabled, the file descriptor and SCM_RIGHTS of the `user_events_data` file must be sent through the optional continuation stream as [described](#passing_file_descriptor). The runtime will register tracepoints based on the provider configurations passed in, and runtime events will be written directly to the `user_events_data` file descriptor. The allow/deny list of Event IDs will apply after the keyword/level filter to determine whether or not that provider's event will be written. +The `CollectTracing5` command is an extension of the `CollectTracing4` command. It has all the capabilities of `CollectTracing4` and introduces new fields to enable a Linux-only user_events-based eventpipe session and to prescribe an enable/disable list for Event IDs. When the user_events-based eventpipe session is enabled, the file descriptor and SCM_RIGHTS of the `user_events_data` file must be sent through the optional continuation stream as [described](#passing_file_descriptor). The runtime will register tracepoints based on the provider configurations passed in, and runtime events will be written directly to the `user_events_data` file descriptor. The enable/disable list of Event IDs will apply after the keyword/level filter to determine whether or not that provider's event will be written. > Note available for .NET 10.0 and later. @@ -678,11 +678,11 @@ The `streaming_provider_config` is composed of the following data: * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider * `string arguments`: (Key-value pairs string to pass to the provider) or (length = 0) -* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). +* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an enable/disable list or (length = `0`). An `event_filter` is comprised of the following data: -* `bool allow`: 0 for deny list, 1 for allow list -* `array event_ids`: List of Event IDs to deny or allow. +* `bool enable`: 0 to disable events, 1 to enable events +* `array event_ids`: List of Event IDs to disable or enable. See [event_filter serialization examples](#event_filter) @@ -696,12 +696,12 @@ The `user_events_provider_config` is composed of the following data: * `uint logLevel`: The level of information to turn on * `string provider_name`: The name of the provider * `string arguments`: (Key-value pairs string to pass to the provider) or (length = 0) -* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an allow/deny list or (length = `0`). +* `event_filter filter`: Rules for filtering this provider's Event IDs, applied after `keyword`/`logLevel`, using an enable/disable list or (length = `0`). * `tracepoint_config config`: Maps Event IDs to tracepoints. If an Event ID is excluded by `event_filter`, it will not be written to any tracepoint. An `event_filter` is comprised of the following data: -* `bool allow`: 0 for deny list, 1 for allow list -* `array event_ids`: List of Event IDs to deny or allow. +* `bool enable`: 0 to disable events, 1 to enable events +* `array event_ids`: List of Event IDs to disable or enable. See [event_filter serialization examples](#event_filter) @@ -736,8 +736,8 @@ A User_events Session started with `CollectTracing5` expects the Optional Contin ### Event_filter Example `event_filter` serialization. Serializing ``` -allow=0, event_ids=[] -Deny Nothing === Allow all events. +enable=0, event_ids=[] +Disable Nothing === Enable all events. ```
logLevel provider_name length provider_name stringfilter_data lengtharguments length
"DOTNET_IPC_V1"
@@ -751,7 +751,7 @@ Deny Nothing === Allow all events. - + @@ -761,8 +761,8 @@ Deny Nothing === Allow all events.
allowenable event_ids
``` -allow=0, event_ids=[4, 5] -Deny only Event IDs 4 and 5 === Allow all Event IDs except 4 and 5 +enable=0, event_ids=[4, 5] +Disable only Event IDs 4 and 5 === Enable all Event IDs except 4 and 5 ``` @@ -777,7 +777,7 @@ Deny only Event IDs 4 and 5 === Allow all Event IDs except 4 and 5 - + @@ -789,8 +789,8 @@ Deny only Event IDs 4 and 5 === Allow all Event IDs except 4 and 5
array<uint>
allowenable event_ids
``` -allow=1, event_ids=[] -Allow Nothing === Deny all events. +enable=1, event_ids=[] +Enable Nothing === Disable all events. ``` @@ -805,7 +805,7 @@ Allow Nothing === Deny all events. - + @@ -815,8 +815,8 @@ Allow Nothing === Deny all events.
allowenable event_ids
``` -allow=1, event_ids=[1, 2, 3] -Allow only EventIDs 1, 2, and 3 === Deny all EventIDs except 1, 2, and 3. +enable=1, event_ids=[1, 2, 3] +Enable only EventIDs 1, 2, and 3 === Disable all EventIDs except 1, 2, and 3. ``` @@ -834,7 +834,7 @@ Allow only EventIDs 1, 2, and 3 === Deny all EventIDs except 1, 2, and 3. - + @@ -854,7 +854,7 @@ Output_format=1, encode bytes for tracepoint_config ``` ``` -All allowed Event IDs will be written to a default "MyTracepoint" tracepoint +All enabled Event IDs will be written to a default "MyTracepoint" tracepoint ```
allowenable event_ids
@@ -879,8 +879,8 @@ All allowed Event IDs will be written to a default "MyTracepoint" tracepoint
``` -Allowed Event IDs 1 - 9 will be written to tracepoint "LowEvents". -All other allowed Event IDs will be written to "MyTracepoint" +Enabled Event IDs 1 - 9 will be written to tracepoint "LowEvents". +All other enabled Event IDs will be written to "MyTracepoint" ``` @@ -935,8 +935,8 @@ All other allowed Event IDs will be written to "MyTracepoint"
``` -Allowed Event IDs 1 - 9 will be written to tracepoint "LowEvents". -No default tracepoint needed, don't write any other allowed Event IDs +Enabled Event IDs 1 - 9 will be written to tracepoint "LowEvents". +No default tracepoint needed, don't write any other enabled Event IDs ``` From eda74d6a90fbc84d265882d68d55faad457466f3 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Wed, 4 Jun 2025 12:16:01 -0400 Subject: [PATCH 12/14] Add bool to general payload encoding spec and clarify description --- documentation/design-docs/ipc-protocol.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index 8e005bfc7e..8ce69dfaf6 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -192,6 +192,7 @@ Payloads are either encoded as fixed size structures that can be `memcpy`'ed , _ * `uint` = 4 little endian bytes * `ulong` = 8 little endian bytes * `wchar` = 2 little endian bytes, UTF16 encoding +* `bool` = 1 unsigned byte * `array` = uint length, length # of `T`s * `string` = (`array` where the last `wchar` must = `0`) or (length = `0`) @@ -355,8 +356,8 @@ EventPipe Payloads are encoded with the following rules: * `uint` = 4 little endian bytes * `ulong` = 8 little endian bytes * `wchar` = 2 little endian bytes, UTF16 encoding -* `byte` = 1 unsigned little endian byte -* `bool` = 1 unsigned little endian byte +* `byte` = 1 unsigned byte +* `bool` = 1 unsigned byte * `array` = uint length, length # of `T`s * `string` = (`array` where the last `wchar` must = `0`) or (length = `0`) From ee73c082375954040e623e87e0585db68bc6a587 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Wed, 4 Jun 2025 12:16:55 -0400 Subject: [PATCH 13/14] Rename output_format to session_type Parallel renaming to Runtime counterpart PR. The serialization format and output format were deemed confusing to have side by side, so renamed the `output format` to more clearly represent its usage --- documentation/design-docs/ipc-protocol.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index 8ce69dfaf6..503ec361b9 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -667,7 +667,7 @@ The `CollectTracing5` command is an extension of the `CollectTracing4` command. Header: `{ Magic; 20 + Payload Size; 0x0206; 0x0000 }` #### Streaming Session Payload: -* `uint output_format`: 0 +* `uint session_type`: 0 * `uint streaming_circularBufferMB`: Specifies the size of the Streaming session's circular buffer used for buffering event data. * `uint streaming_format`: 0 for the legacy NetPerf format and 1 for the NetTrace V4 format. Specifies the format in which event data will be serialized into the IPC Stream * `ulong rundownKeyword`: Indicates the keyword for the rundown provider @@ -688,7 +688,7 @@ An `event_filter` is comprised of the following data: See [event_filter serialization examples](#event_filter) #### User_events Session Payload: -* `uint output_format`: 1 +* `uint session_type`: 1 * `ulong rundownKeyword`: Indicates the keyword for the rundown provider * `array providers`: The providers to turn on for the session @@ -850,8 +850,8 @@ Enable only EventIDs 1, 2, and 3 === Disable all EventIDs except 1, 2, and 3. ### Tracepoint_config Example `tracepoint_config` serialization ``` -Output_format=0, Streaming Sessions DO NOT encode bytes for tracepoint_config -Output_format=1, encode bytes for tracepoint_config +session_type=0, Streaming Sessions DO NOT encode bytes for tracepoint_config +session_type=1, encode bytes for tracepoint_config ``` ``` @@ -993,7 +993,7 @@ No default tracepoint needed, don't write any other enabled Event IDs > Note: This only applies to enabling an user_event-based EventPipe session, which is specifically a Linux feature -To register [user_event](https://docs.kernel.org/trace/user_events.html) tracepoints and write events, access to the root protected `user_events_data` file is required. Once the .NET Runtime's Diagnostic Server processes a [CollectTracing5](#collecttracing5) command specifying the `user_events` format (`output_format=1`), it expects that the client will send a file descriptor to the [continuation stream](#general-flow) via SCM_RIGHTS. +To register [user_event](https://docs.kernel.org/trace/user_events.html) tracepoints and write events, access to the root protected `user_events_data` file is required. Once the .NET Runtime's Diagnostic Server processes a [CollectTracing5](#collecttracing5) command specifying the `user_events` format (`session_type=1`), it expects that the client will send a file descriptor to the [continuation stream](#general-flow) via SCM_RIGHTS. ```C #include From 432f6b81201d5b01a8b05175dddaf1a026830396 Mon Sep 17 00:00:00 2001 From: mdh1418 Date: Tue, 10 Jun 2025 23:38:42 -0400 Subject: [PATCH 14/14] Move metadata into extensions and document format --- documentation/design-docs/ipc-protocol.md | 41 ++++++++++++++++++++--- 1 file changed, 36 insertions(+), 5 deletions(-) diff --git a/documentation/design-docs/ipc-protocol.md b/documentation/design-docs/ipc-protocol.md index 503ec361b9..7e64348d86 100644 --- a/documentation/design-docs/ipc-protocol.md +++ b/documentation/design-docs/ipc-protocol.md @@ -1027,7 +1027,7 @@ Once the runtime has received the configured tracepoint names as detailed under ##### Tracepoint Format V1 -` u8 version; u16 event_id; __rel_loc u8[] extension; __rel_loc u8[] payload; __rel_loc u8[] meta` +` u8 version; u16 event_id; __rel_loc u8[] extension; __rel_loc u8[] payload` See [user_events writing](#user_events-writing) below for field details`. @@ -1044,12 +1044,10 @@ io[1].iov_base = &version; // __u8 tracepoint format version io[1].iov_len = sizeof(version); io[2].iov_base = &event_id; // __u16 EventID defined by EventSource/native manifest io[2].iov_len = sizeof(event_id); -io[3].iov_base = &extension; // __rel_loc u8[] NetTrace V6 label list +io[3].iov_base = &extension; // __rel_loc u8[] optional event information io[3].iov_base = sizeof(extension); io[4].iov_base = &payload; // __rel_loc u8[] event payload io[4].iov_len = sizeof(payload); -io[5].iov_base = &meta; // __rel_loc u8[] event metadata -io[5].iov_len = sizeof(meta); io[6].iov_base = &data; // __u8[] data io[6].iov_len = data_len; @@ -1064,7 +1062,40 @@ The `version` is the version of the tracepoint format, which in this case is [ve The `event_id` is the ID of the event, defined by the EventSource/native manifest. -The `extension` points at a [NetTrace V6 LabelList block](https://github.com/microsoft/perfview/blob/main/src/TraceEvent/EventPipe/NetTraceFormat.md#labellistblock) describing other fields associated with the event. e.g. ActivityId, RelatedActivityId, event_thread, and stack. +#### Extension Blob Format + +The `extension` field is an optional data blob that can provide additional information about an event. Its structure is as follows: + +1. **Label** (`byte`): Indicates the type of data that follows. +2. **Data**: The content, whose format depends on the label. + +**Label Values and Corresponding Data:** + +| Label | Meaning | Data Format | Description | +|--------|------------------------|----------------------------|-----------------------------------------------------------------------------| +| 0x01 | Event Metadata | `array metadata` | Contains event metadata, formatted per NetTrace v5. | +| 0x02 | ActivityId | `uint16 guid` | Contains the GUID for the ActivityId. | +| 0x03 | RelatedActivityId | `uint16 guid` | Contains the GUID for the RelatedActivityId. | + +**Details:** +- The extension blob may be empty if no extra information is present. +- Multiple extension blobs can be concatenated if more than one piece of information is needed. Each blob starts with its own label byte. +- For Event Metadata (`0x01`), the `metadata` array matches the NetTrace v5 PayloadBytes format. +- For ActivityId and RelatedActivityId (`0x02`, `0x03`), the `guid` is a 16-byte value representing the GUID. +- The size of the entire extension blob can be inferred from the extension `__rel_loc` field. See the [__rel_loc documentation](https://lwn.net/Articles/876682/) for more details. + +**Example Layout:** + +``` +[Label][Data][Label][Data]... +``` + +For example, an extension blob containing both Event Metadata and ActivityId would look like: +- `[0x01][metadata][0x02][guid]` + +**Notes:** +- The runtime includes Event Metadata only the first time an event is sent in a session. +- Native runtime events do not include metadata. The `payload` points at a blob of data with the same format as an EventPipe payload – the concatenated encoded values for all the parameters.