[API Proposal]: Expose TLS signature algorithm families for server certificate selection #134630

Background and motivation

During a post-quantum cryptography rollout, servers need to keep classical certificates available while selectively offering ML-DSA certificates to compatible clients. Today, ServerOptionsSelectionCallback can select a certificate based on SNI and advertised TLS versions, but SslClientHelloInfo does not indicate which signature algorithm families the client advertised.

Selecting an ML-DSA certificate unconditionally breaks classical clients, while always selecting RSA or ECDSA prevents gradual PQC deployment on the same endpoint. The same SNI may therefore need to select among RSA, ECDSA, and ML-DSA certificates according to the client's capabilities and the server's preference.

This proposal exposes a compact family-level view of the ClientHello signature_algorithms extension. It is intended as an allocation-free pre-filter for choosing among configured certificate candidates. It deliberately does not claim that every certificate or chain in a reported family is compatible; the application must still apply its certificate policy and retain an appropriate fallback.

TLS signature algorithms are independent from TLS supported groups:

• Signature algorithms determine certificate and handshake-signature compatibility.
• Supported groups determine key exchange, including hybrid ML-KEM groups.

Negotiated supported-group reporting is tracked separately by #132239. Additional PQC SslStream test coverage is tracked by #134227.

The existing low-level workaround is to use the experimental TlsSession raw ClientHello API and implement a TLS parser. Normal SslStream and Kestrel callback users cannot otherwise inspect this information.

API proposal

namespace System.Net.Security;

[Flags]
public enum TlsSignatureAlgorithmFamilies
{
    None = 0,
    Rsa = 1,
    ECDsa = 2,
    EdDsa = 4,
    MLDsa = 8,
    SlhDsa = 16,
}

public readonly partial struct SslClientHelloInfo
{
    // Existing
    // public SslClientHelloInfo(string serverName, SslProtocols sslProtocols);

    public SslClientHelloInfo(
        string serverName,
        SslProtocols sslProtocols,
        TlsSignatureAlgorithmFamilies signatureAlgorithmFamilies);

    public TlsSignatureAlgorithmFamilies SignatureAlgorithmFamilies { get; }
}

SignatureAlgorithmFamilies reports recognized families from the client's signature_algorithms extension. None means that the information is unavailable or that no recognized family was advertised.

The property is intentionally a broad compatibility signal:

• MLDsa means the client advertised at least one recognized ML-DSA signature scheme.
• It does not mean the client supports every ML-DSA parameter set.
• It does not guarantee compatibility with every certificate chain using that family.
• The server controls certificate preference ordering.

API usage

await sslStream.AuthenticateAsServerAsync(
    (stream, hello, state, cancellationToken) =>
    {
        SslStreamCertificateContext certificate =
            (hello.SignatureAlgorithmFamilies & TlsSignatureAlgorithmFamilies.MLDsa) != 0
                ? mlDsaCertificate
                : (hello.SignatureAlgorithmFamilies & TlsSignatureAlgorithmFamilies.ECDsa) != 0
                    ? ecdsaCertificate
                    : rsaCertificate;

        return ValueTask.FromResult(new SslServerAuthenticationOptions
        {
            ServerCertificateContext = certificate,
        });
    },
    state: null,
    cancellationToken);

Applications selecting a specific ML-DSA parameter set, RSA-PSS key form, or ECDSA curve would need more detailed information than this proposal provides. Such applications should retain a compatible fallback.

Alternative designs

Expose exact signature schemes

public ReadOnlyMemory<TlsSignatureScheme> SignatureAlgorithms { get; }
public ReadOnlyMemory<TlsSignatureScheme> CertificateSignatureAlgorithms { get; }

This is lossless and can distinguish:

• ML-DSA parameter sets.
• RSA-PSS RSAE versus RSA-PSS-constrained keys.
• ECDSA curve and hash constraints.
• Certificate-chain signature constraints.
• Client ordering and unknown schemes.

However, it exposes substantially more TLS protocol detail, requires callers to implement family mapping and signature_algorithms_cert fallback semantics, and requires storage proportional to the advertised lists. The proposed bitmask addresses the common certificate-family rollout policy without adding collection allocations.

Exact schemes could be exposed instead of, or later in addition to, the family-level property if API review determines that parameter-set selection is the primary scenario.

Expose both signature_algorithms and signature_algorithms_cert as family masks

signature_algorithms_cert constrains signatures within the certificate chain, not simply the leaf key family. For example, an ECDSA leaf can be issued by an RSA CA. Reducing that extension to a second family mask could encourage callers to incorrectly require the leaf family in both masks.

The proposal instead documents that the family mask is only a broad leaf/private-key compatibility signal.