bug: browser RSA key generation exposes raw JavaScript errors for out-of-range modulusLength

[harrshita123]

Description

The browser backend forwards RSA modulusLength values directly to SubtleCrypto.generateKey.

RsaKeyGenParams.modulusLength is defined as an [EnforceRange] unsigned long. Values outside 0..0xffffffff fail WebIDL conversion and surface through Dart2JS as a raw JSObject: TypeError instead of a package-owned Dart exception.

This affects:

• RsaOaepPrivateKey.generateKey
• RsaPssPrivateKey.generateKey
• RsassaPkcs1V15PrivateKey.generateKey

Reproduction

await RsaOaepPrivateKey.generateKey(
  -1,
  BigInt.from(65537),
  Hash.sha256,
);

await RsaOaepPrivateKey.generateKey(
  0x100000000,
  BigInt.from(65537),
  Hash.sha256,
);

Actual behavior

Chrome with Dart2JS exposes:

JSObject: TypeError: Failed to execute 'generateKey' on 'SubtleCrypto':
RsaHashedKeyGenParams: modulusLength: Outside of numeric range

The native backend rejects the same inputs using a Dart UnsupportedError.

Expected behavior

Out-of-range values should be rejected with a package-owned Dart exception before browser interop. Raw JavaScript exceptions should not escape from the public API.

Suggested fix

• Validate the WebIDL unsigned-long range at the shared public API boundary.
• Apply the validation to all three RSA key-generation APIs.
• Add cross-backend regression tests for -1 and 0x100000000.
• Assert the exact Dart exception type on VM and browser backends.

Specification

The Web Cryptography specification defines RsaKeyGenParams.modulusLength as a required [EnforceRange] unsigned long:

https://www.w3.org/TR/WebCryptoAPI/#dfn-RsaKeyGenParams