Repository navigation
Build ID Section for WASM #133
Description
I originally brought this up in the design repo (WebAssembly/design#1306) but I believe this fits here better.
For deferred symbolication on services like sentry it would be nice to be able to match up DWARF debug information to the main WASM file by build ID. In ELF this is typically accomplished with the GNU build ID note, on windows with the PDB signature and age and on darwin the macho UUID fulfills that purpose.
I would love to see a build_id custom section that contains a 16 or 20 byte ID which tools would ensure remains in both WASM files (CODE, debug companion containing DWARF info) if they get split. Capping it at 16 bytes makes it possible to roundtrip this through breakpad which uses a 16+4 byte char array for the debug id. 16 for the PDB UUID + 4 byte for the PDB age.
Motivation: Sentry and other systems like to be able to look up files by build ID because then they can access an external symbol server for that information. That way one just provides some sources where debug information can be found and then symbolicators just reach out to that service to find the debug information files.
Activity
kripken commentedon Nov 14, 2019
Sounds good to me.
Would the build ID be modified by tools that process the wasm, say by Binaryen when it optimizes the binary? Or would it stay fixed after it's emitted from the original compiler? (The former seems to make sense, as optimizations change the binary, but then we'd need to describe those changes here I think.)
mitsuhiko commentedon Nov 14, 2019
Definitely should change as the file changes.
Of note is that in the Microsoft ecosystem the age on the PDB signature (those extra 4 bytes) get incremented with every transformation. This from my experience has made things more complicated in practice because they were not consistently changed everywhere. For instance the age is stored more than once in the PE format and actually comes out desynched from Microsoft's own tools.
I think it would be wiser to explicitly tell tools to always completely override the embedded ID if it goes through a transformation. This does mean you can't track back to the original ID of the originally created WASM file but I'm not sure if that is necessary in general.
Would be curious to hear though if there are some advantages of the pdb+age system on the Microsoft side.
sunfishcode commentedon Nov 14, 2019
How important is it to have an explicit field for this, as opposed to just having tools compute a hash of a wasm binary to use as an effective build ID?
mitsuhiko commentedon Nov 14, 2019
@sunfishcode since the ID needs to survive a stripping of the file, it's very important. With DWARF in place you normally want to separate out the object file into two: one that contains CODE and other sections necessary to run the code, a second one with the DWARF sections (
.debug_frameetc.). Since stripping/splitting the files changes the file you cannot reproduce the ID after the split.sunfishcode commentedon Nov 14, 2019
Can tools just hash the contents of the main wasm sections then, and ignore debug info sections?
I don't have a strong opinion either way yet; I just want to understand the space.
mitsuhiko commentedon Nov 14, 2019
To generate the build ID they could take the hash of the main wasm sections and store it in the file. They can alternatively just generate a random UUID and embed it. I do think though that a build ID should ideally always be embedded.
(This here describes the workflow where this information is particularly useful)
sunfishcode commentedon Nov 15, 2019
I'm curious about what situations storing a Build ID in the file is better than computing a hash on demand whenever it's needed.
Naively, computing it on demand would seem to have several advantages:
Jake-Shadle commentedon Nov 15, 2019
FWIW Breakpad supports the embedded, format specific identifiers, that @mitsuhiko mentioned, but if they aren't available for any reason, it falls back to computing an md5 of the first 1024 bytes of the TEXT section (or equivalent).
You bring up some good points about some advantages computing the build ID has, but to me the point of the Build ID is to precisely identify a particular build so that different tools can always pair the code with the debug information, so allowing tools to choose their own hash function or which sections to hash, brings up problems when tools need to communicate with each other, eg. between a debugger and a symbol store that use different hash functions.
So storing the Build ID does have some disadvantages, particularly when a tool does a transformation that doesn't also change the Build ID, but I'm much more concerned with tools having a consistent source of truth.
sunfishcode commentedon Nov 15, 2019
I've now found this stackoverflow post which I found helpful. The Build ID isn't just a hash of the contents; it's something like a hash of the contents and the debug info together, which is then recorded and preserved, even if debug info is stripped. As such, it can't always be recomputed.
There are a lot of use cases other than debug info that would seem to want something like a Build ID, but what they need is something subtly different from what the Build ID actually is. So, brainstorming here, what if we do have a Build ID section, but call it the "Debug Info ID", and say:
Would that make sense?
mitsuhiko commentedon Nov 15, 2019
I think it's fair to specifically call this a
debug_idand record that it's useful for that purpose.The hashing fallback path of breakpad has caused more issues than it solved so I would prefer we don't spec out something like this.
sbc100 commentedon Nov 15, 2019
Would this build ID be generated at the point when the debug info is split out (either by the linker, or some kind of post link debug-splitting tool)? Or would it be present even in binaries that still have their debug info embedded?
mitsuhiko commentedon Nov 15, 2019
@sbc100 definitely already in binaries that have the debug info embedded. We for instance have lots of cases where we want to symbolicate stacktraces where the client just submitted instruction addresses and then people upload the entire binary with debug information included.
This is especially important normally when doing stack unwinding out of memory dumps. This obviously is less useful for wasm right now, but in terms of existing work flows having the debug ID even in unstripped binaries has been very valuable.
RReverser commentedon Feb 13, 2020
Just as a counter-point, one downside of an embedded id seems to be precisely that it would usually survive destructive operations on the code.
That is, if code is post-processed by a tool similar to wasm-opt or wasm-bindgen, and if that tool can't correctly update DWARF information, then the build id would remain the same even though the code has changed and no longer matches the debug info. In this case you as a consumer (Sentry or otherwise) explicitly don't want such debug info to be matched and used.
Arguably, every such tool should either support DWARF or be able to at least change build ID to some new unique value, but it seems that hashing of code section would alleviate this concern even more naturally.
mitsuhiko commentedon Nov 24, 2020
Since we're adding WASM DWARF support at Sentry at the moment we might be going ahead and require customers to embed a
build_idcustom section into their files for now.RReverser commentedon Nov 25, 2020
@mitsuhiko Does the "hash of the code section" idea not work for you?
31 remaining items
dschuff commentedon Jun 11, 2022
@mitsuhiko I guess a followup question, if I were to make lld generate a v5 UUID (based on, a hash of the contents), what would I use as the "namespace" UUID to go with it?
bkotsopoulossc commentedon Jun 13, 2022
Would it be reasonable to just generate a random UUID once and bake it into the llvm code, as an "llvm namespace"?
mitsuhiko commentedon Dec 3, 2022
@dschuff about the namespace it probably doesn't matter. You can probably hardcode a random ID and just use that consistently and document it. I don't have any expectations that there is a tool independent way of generating the same reproducible IDs. It's more important that the tool itself has some stability.
dschuff commentedon Feb 22, 2023
I updated the prototype in https://reviews.llvm.org/D107662
It supports several different styles for compatibility (mostly the same ones as ELF).
The default style ("fast" aka "tree") hashes the contents of the output and (unlike ELF) generates a v5 UUID based on the hash (using a random namespace).
It also supports generating a random v4 UUID, a sha1 hash, and a user-specified string (as ELF does).
dschuff commentedon Mar 2, 2023
I think the implementation and document in #183 capture what we've worked out here. Feel free to reopen (or open a new issue, as appropriate) if there are objections or changes we should make; Since there's no ecosystem yet I don't think it's too late to make breaking changes if we do it soon.
bkotsopoulossc commentedon Mar 6, 2023
This is awesome, I see that this just landed in llvm, can you update once we know the version of emscripten this is associated with?
dschuff commentedon Mar 9, 2023
This change is now included in emscripten 3.1.33
jedisct1 commentedon Mar 9, 2023
The Zig wasm linker supports it as well.
sbc100 commentedon Mar 9, 2023
Zig has its own wasm linker? Is it based on wasm-ld or something different?
jedisct1 commentedon Mar 9, 2023
Written from scratch, like other linkers.
bkotsopoulossc commentedon Mar 31, 2023
@mitsuhiko should we update the sentry docs? https://github.com/getsentry/sentry-docs/blob/61d1ccd322634f845bb89b0e5243c976f86e114c/src/platforms/common/data-management/debug-files/identifiers.mdx?plain=1#L89
RReverser commentedon Mar 31, 2023
FWIW this tag is now also natively recognised by
wasmbin(used by Sentry'swasm-split):CustomSection::BuildIdAdd a document for build IDs (WebAssembly#183)
[debug][wasm] Add support for 'build_id' custom section