Add _promise variants for __async library functions

[guybedford]

Lets one asynchronous JS library body be called from any stack, by also exposing it as a C function returning a promise of its result.

__async can now also be a list of variants, as __async: ['promise'], which otherwise behaves as 'auto'. The promise variant foo_promise runs the same body (with sync always false), returning an em_promise_t on the calling thread, fulfilled with the result (NULL for a void function) or rejected with NULL. It is emitted only where used. Since variants are listed explicitly, only those C symbols exist: for system libraries each one gets a header declaration, and gen_sig_info handles it like any other function. From a pthread it is emscripten_proxy_promise_with_ctx of the main-thread body, with the result carried in the proxied call's arg and a then on the completion promise handing it over; the pthread-side wrapper is a plain call returning the handle. This has its own entry point so nothing is linked into pthread builds that don't use it. proxying.h is unchanged.

lookup__sig: 'pp', lookup__async: ['promise'], lookup__proxy: 'sync',
lookup: (name, sync) => sync ? -EAGAIN : fetchSomething(name),

intptr_t lookup(const char* name);
em_promise_t lookup_promise(const char* name);

Docs are added to the existing "Marking JS library functions as async" section.

Tests: jslib.test_jslib_async_auto_variants with pointer-returning and void functions in plain, ASYNCIFY, JSPI and PROXY_TO_PTHREAD builds, wasm32 and wasm64, and jslib.test_jslib_async_variants_errors for invalid variant lists.

Made with AI assistance under my review

[tlively]

Instead of building the _promise version of a function on top of the _fd version, I would expect the _promise version to be implemented in terms of emscripten_proxy_promise_with_ctx and the _fd version to be implemented in terms of emscripten_proxy_fd_with_ctx. That does not yet exist, but it would look very much like what you have in system/lib/libc/emscripten_fd_promise.c.

This would avoid the overhead of the FD system for the (hopefully) more common case of using promising calls, and I hope it would simplify the code as well.

[guybedford]

I would expect the _promise version to be implemented in terms of emscripten_proxy_promise_with_ctx and the _fd version to be implemented in terms of emscripten_proxy_fd_with_ctx.

@tlively thanks for taking a look, I've added a direct implementation now for the _promise version, using emscripten_proxy_callback_with_ctx. One issue was that it doesn't currently support passing result data so I've added a new proxy_finish_with_result that allows passing a pointer sized result data. Let me know if you'd rather I split that out separately.

For the _fd version, having an emscripten_proxy_fd_with_ctx, isn't necessary because descriptors are already shared, and all operations using them already are __proxy: 'sync'.

For now the new proxy attributes are fully manual - __proxy: 'fd' and __proxy: 'promise' as the attributes.

[sbc100]

Can we now perhaps split out the _promise version so we can separately consider the whole fd thing? I like the idea of landing the mechanism for multiple-flavors separately to committing the fb mechanism.

Would, since we considering all the different falvors here, would it be worth also adding an auto-generated _callback flavor? Then the result could be delivered directly to user who just want a callback.

[guybedford]

Can we now perhaps split out the _promise version so we can separately consider the whole fd thing? I like the idea of landing the mechanism for multiple-flavors separately to committing the fb mechanism.

Sure, I've split out the fd version from this PR.

Would, since we considering all the different falvors here, would it be worth also adding an auto-generated _callback flavor? Then the result could be delivered directly to user who just want a callback.

A __proxy: 'callback' could definitely make sense as another follow-on variant like the fd one. It would be the first to actually change the C signature.

[sbc100]

This extra exported metadata is only needed for gen_sig_info.py, IIUC?

I wonder if we can find a way to avoid it? And this avoid changing the JSON API between Python and JS compiler? At least maybe add a comment, maybe a TODO to remove it later?

[guybedford]

Variants have to be in the --symbols-only symbol graph, since that's what wasm-ld receives as the allowed JS imports, so that answer_promise links at all. With automatic expansion, that graph also contained variants of internal system functions (_emval_await_promise, _dlopen_js_promise, emscripten_promise_await_unchecked_promise). None of those have a header, so gen_sig_info needed the metadata to skip them.

Instead of force adding these headers for all async auto cases, I've actually refactored back again to a non-implicit convention in contrast to previous reviews for an __async: ['promise'] explicit variation system. This way, only listed variants now exist - for system libraries each one gets a header declaration and then gen_sig_info handles it normally.

I'm open to reconsidering the auto expansion, but we'd need to have a plan to avoid the header requirements on existing system: 'auto' APIs that we don't want to maintain these variant APIs for.

[guybedford]

Unless you want to just add C headers for all these system variants?

[sbc100]

Indeed _dlopen_js exists in system/lib/libc/emscripten_internal.h already so it think it makes sense to put _dlopen_js_promise alongside it, no? WDYT? I think its kind of nice that all callable JS functions are declared in some header.

[guybedford]

This means we would have _dlopen_js_promise as well as emscripten_dlopen_promise. If we're fine with that we can do that, it's more just a question of whether we want to do this for all auto functions now, or go back to the opt-in system? I haven't fully gone through all the 27 system functions to determine if they all make sense to be defined in the headers yet. Another one that doesn't really make sense is emscripten_promise_await_unchecked_promise - a promise of awaiting a promise.

[guybedford]

@sbc100 I separated out the first layer of this adding the sync arg to async functions in #28022 as the new bottom of the stack.

In terms of the driving use cases:

• Base PR Pass sync to __async: 'auto' library functions #28022 adding sync arg: allows for blocking sockets calls, as well as allowing the async variants here to coexist with the sync variants
• This PR Add _promise variants for __async library functions #27880: lays the groundwork for a function to be both sync and async, with DNS as the canonical example carried at the top of the stack.

I've included the callback variant in the stack, while removing the fd variant from the stack for now. Starting from the bottom towards these use cases would continue to make sense.