mcp: Implement OAuth 2.1 Grant Flows (Authorization Code + PKCE and Client Credentials)

[vblagoje]

Scope

Implement both OAuth 2.1 grant types per RFC 6749 and MCP 2.1 spec:

• Authorization Code + PKCE (RFC 7636): For user-delegated access (interactive, requires browser)
• Client Credentials (RFC 6749 §4.4): For machine-to-machine access (non-interactive, no user consent)

Haystack MCP Integration Details

OAuthConfig Design:

• Add oauth_config: Optional[OAuthConfig] field to StreamableHttpServerInfo and SSEServerInfo
• OAuthConfig must support to_dict()/from_dict() for serialization (using Haystack's Secret handling)
• Token storage should be configurable (InMemoryTokenStorage or FileTokenStorage)
• Default to FileTokenStorage for persistence across tool instances

Flow Selection Logic:

• Auto-detect grant type based on configuration:
  • If client_secret is provided AND no redirect_uri → use Client Credentials
  • Otherwise → use Authorization Code + PKCE (default)
• Allow explicit grant_type parameter for manual override
• Validate that Client Credentials flow doesn't require redirect_uri

Token Injection:

• OAuth tokens should be injected into server_info.headers["Authorization"] (takes precedence over token field)
• This ensures compatibility with existing StreamableHttpClient and SSEClient header handling
• Token should be cached and reused across tool invocations

Integration with Connection Lifecycle:

• OAuth should be checked in ensure_oauth(server_info) function
• Called from:
  • MCPTool.__init__() if eager_connect=True
  • MCPTool.warm_up() if eager_connect=False
  • MCPToolset._connect_and_load_tools() (always, before connection)
• OAuth flow should be synchronous (blocking) since it's called from sync context
• Use AsyncExecutor for any async OAuth operations if needed

Requirements

• Create OAuthConfig dataclass with:
  • client_id and client_secret (optional, Secret-compatible)
  • redirect_uri (for Authorization Code flow, default: http://localhost:8080/callback)
  • grant_type (optional, auto-detected if not specified)
  • scopes (optional list)
  • token_storage (default: FileTokenStorage)
  • on_authorization_url callback hook (for customizing auth URL presentation)
  • authorization_server_url (optional, for skipping discovery)
  • authorization_endpoint and token_endpoint (optional, for manual configuration)
• Implement Authorization Code + PKCE flow:
  • Generate PKCE code verifier and S256 challenge (RFC 7636)
  • Build authorization URL with required parameters (response_type=code, code_challenge, code_challenge_method=S256)
  • Exchange authorization code for access token (with code_verifier)
  • Handle token response and extract access_token, refresh_token, expires_in
  • Support resource parameter in token request (per MCP spec)
• Implement Client Credentials flow:
  • Direct token exchange using client_id and client_secret
  • POST to token_endpoint with grant_type=client_credentials
  • No user interaction required
  • Handle token response
  • Support resource parameter in token request (per MCP spec)
• Create token storage interface:
  • TokenStorage abstract base class
  • InMemoryTokenStorage (tokens lost on process exit)
  • FileTokenStorage (persist to ~/.mcp/tokens/ with secure permissions)
  • Support for token expiration checking
• Implement ensure_oauth(server_info: MCPServerInfo) function:
  • Check if oauth_config is present and transport is HTTP
  • Check for cached valid token
  • Trigger appropriate OAuth flow if needed
  • Inject token into server_info.headers["Authorization"]
• Add OAuthConfig to StreamableHttpServerInfo and SSEServerInfo:
  • Add oauth_config: Optional[OAuthConfig] = None field
  • Update to_dict()/from_dict() to handle OAuthConfig serialization
  • Ensure Secret objects in OAuthConfig are properly serialized
• Add comprehensive tests for both flows
• Document when to use each flow with examples

References

• MCP Authorization Spec - Grant Types
• OAuth 2.1 Client Credentials Grant (RFC 6749 §4.4)
• OAuth 2.1 Authorization Code Grant with PKCE (RFC 7636)

[ayeshachohan]

Hi — I'd like to take this on, but want to propose splitting the scope before starting.

The issue bundles two grant types (Authorization Code + PKCE, and Client Credentials) plus a token-storage abstraction into one PR. Authorization Code + PKCE also depends on the local-callback-server work in #2547 to be usable end-to-end, so building both grant types together means the PR can't be reviewed or merged independently of that follow-up issue.

Proposed first PR — Client Credentials only (RFC 6749 §4.4): it's non-interactive (no browser/local server dependency), so it's usable standalone. Scope:

• OAuthConfig dataclass with to_dict()/from_dict(), following the Secret/SecretType convention already used for MCPServerInfo.token and headers.
• Add oauth_config: Optional[OAuthConfig] to StreamableHttpServerInfo and SSEServerInfo.
• InMemoryTokenStorage now, with the interface shaped so FileTokenStorage can be added in a follow-up without breaking callers.
• Token acquisition + refresh wired into the existing header-resolution path (_resolve_headers).

Authorization Code + PKCE would follow once #2547's callback server lands, since it needs it to be usable. Let me know if you'd rather I scope this differently.