Verify SSH host keys by default in SSH and SFTP hooks

[potiuk]

The SFTP filesystem backend (providers/sftp/.../fs/sftp.py) has always verified host keys by default. SSHHook, SSHHookAsync and SFTPHookAsync did not — no_host_key_check defaulted to true, so any connection that didn't say otherwise got paramiko's AutoAddPolicy, or known_hosts=None on the asyncssh paths. This brings the hooks in line with the filesystem backend.

Breaking change

A connection to a host with no entry in the known hosts file is now refused. Existing behaviour can be kept by any of:

• adding the host key to the known hosts file
• supplying host_key on the connection
• setting the no_host_key_check connection extra to true
• passing no_host_key_check=True to the hook constructor

The constructor argument takes precedence over the connection extra.

Three supporting changes

no_host_key_check constructor argument on SSHHook. The setting could only come from a Connection extra, so callers building the hook directly had no opt-out at all — the migration path above would not have worked for them.

ignore_hostkey_verification is now a deprecated alias. Nothing read this extra. Connections setting it were relying on the permissive default, not on the setting, so flipping the default would have broken them with no working replacement. It now works and emits an AirflowProviderDeprecationWarning.

Worker hooks inherit configuration. store_directory_concurrently and retrieve_directory_concurrently built their per-worker hooks as SFTPHook(ssh_conn_id=self.ssh_conn_id), dropping everything else configured on the parent. Harmless while the default was permissive; now the workers inherit the host key setting.

Test plan

• breeze testing providers-tests --test-type "Providers[ssh,sftp]" — 404 passed, 0 failed
• New tests verified to fail without the source changes (checked by reverting each)
• ruff check, ruff format --check, and the pre-commit hook set all clean

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines at
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions

[potiuk]

Addressed all three points in 292ce203ff and rebased onto current main:

• apache-airflow-providers-ssh in the SFTP package is now marked # use next version, so release preparation raises the minimum and the mixed-install TypeError can't happen.
• ignore_hostkey_verification is now resolved in SSHHookAsync._parse_extras and SFTPHookAsync._parse_extras, canonical key first, same deprecation warning — the sync-submit / async-trigger split in SSHRemoteJobOperator is closed.
• The host_key conflict check now runs on the effective value, after the constructor override, so an explicit no_host_key_check=False resolves a connection that sets both. Connection-only contradictions still raise, and an explicit True against a connection host key is now rejected symmetrically.

Six tests added, two per touched test file. breeze testing providers-tests --test-type "Providers[ssh,sftp]": 410 passed, 7 skipped (404 before). PTAL.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[shubhamraj-git]

Two additional pre-existing issues surfaced while reviewing this change. I do not consider them blockers for this PR, but they are worth tracking:

1. SFTPHookAsync._parse_extras() compares the expanded constructor value with the unexpanded default_known_hosts using !=. As a result, a connection-level known_hosts extra overwrites even an explicit constructor path. A regression test could construct SFTPHookAsync(known_hosts="/explicit/known_hosts"), parse a connection containing {"known_hosts": "/connection/known_hosts"}, and assert that the explicit path wins, consistently with SSHHookAsync.

2. The concurrent SFTP worker hooks now preserve no_host_key_check, but they still discard the parent hook's remote_host, port, username, credentials, key file, proxy configuration, and other constructor overrides. In particular, a directory can be enumerated through an explicit remote_host, after which the workers connect to the original host from the connection. A regression test could patch worker hook construction, configure a parent with a different remote_host, and assert that every worker receives the effective host and relevant connection settings.

---

Drafted-by: Codex (GPT-5); reviewed by @shubhamraj-git before posting

[potiuk]

Addressed the second round in a57753ee30:

• Deferrable SFTPOperator keeps the hook's opt-out. SFTPHookAsync gains a no_host_key_check constructor argument (same precedence and conflict rule as SSHHook), SFTPTransferTrigger carries it (serialized), and the operator passes its hook's effective value when deferring. SFTPHookAsync._get_conn now parses extras unconditionally, so the override also applies to connections without extras. The regression test is your suggested shape: supplied SFTPHook(..., no_host_key_check=True) → TaskDeferred → the trigger's async hook connects with known_hosts=None.
• Effective host in the inline known_hosts entry. Both async hooks now use self.host or conn.host, matching what asyncssh.connect() targets; tests cover the override case on each hook.
• Bare base64 host_key. Normalized to ssh-rsa <key> in both async hooks, consistent with the sync hook. The parametrized SFTP test now uses a valid bare RSA key and asserts the three-field entry.
• Docs wording. "accepting unknown host keys without requiring a known-hosts entry" in both connection docs.
• Test doubles. The three async SSH tests use real Connection objects.

The two pre-existing issues you flagged separately are tracked in #73584 (SFTPHookAsync known_hosts precedence) and #73585 (concurrent worker hooks dropping overrides).

breeze testing providers-tests --test-type "Providers[ssh,sftp]": 471 passed, 7 skipped. PTAL.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[potiuk]

Thanks for digging into these. The first one is already fixed on main by #73593 — _parse_extras now compares against the expanded default, so an explicit constructor known_hosts wins over the connection extra, consistent with SSHHookAsync.

The second is real and still there: the concurrent workers are built from ssh_conn_id alone (plus no_host_key_check after this PR), so any constructor overrides — remote_host, port, username, credentials, key file, proxy — are lost, and a directory listed through an explicit remote_host is downloaded from the connection's host. That's out of scope for this PR, so I've opened #74208 to track it.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[potiuk]

Follow-up for the second issue: #74209 fixes #74208 — the concurrent SFTP workers are now copies of the parent hook, so they keep remote_host, port, credentials, proxy and host key settings instead of being rebuilt from ssh_conn_id alone.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[dabla]

The default flip is consistent across SSHHook, SSHHookAsync, SFTPHook and SFTPHookAsync, and the precedence (constructor argument, then connection extra, then verify) is validated on the effective value in both the sync and the async hooks. CI is green on 30482241ff.

Two new points. The branch conflicts with main in both hook files (#73647 and #74211), and the #73647 side needs more than a mechanical resolution, see comment [1]. And on the async hooks the new default path ends in a FileNotFoundError when the known hosts file does not exist, see comments [2] and [3].

One observation without an inline comment: SSHHookAsync gets no no_host_key_check constructor argument while SFTPHookAsync does. Nothing in the repo needs it (SSHRemoteJobOperator builds both of its hooks from the connection only) and known_hosts="none" still works as an opt-out, so this is an API asymmetry rather than a defect.

I checked the other in-repo consumers of the changed default (Amazon, Google and Azure SFTP transfers, Teradata BTEQ/TPT, SFTPSensor, SFTPClientPool, SSHOperator): all build their hooks from the connection only, so they follow the documented migration path. ComputeEngineSSHHook does not call SSHHook.__init__ and keeps its own host_key_policy, so it is unaffected.

[potiuk]

All comments resolved @dabla

[dabla]

All three points from my review of 30482241ff are addressed on 5355c9a9, and CI is green on the rebased head (71 checks passed, 24 skipped).

• The #73647 conflict is resolved the way I suggested: _build_worker_hook() now passes no_host_key_check=self.no_host_key_check as a constructor argument instead of assigning it afterwards, so a parent built with no_host_key_check=False against a connection that sets both host_key and no_host_key_check=true produces workers that construct without raising. test_concurrent_transfer_passes_effective_host_key_policy_to_worker covers exactly that connection.
• Both async hooks gained _should_use_known_hosts(): when known_hosts is still the expanded default and the file is absent, the key is left out of the asyncssh config, so a triggerer without ~/.ssh/known_hosts gets HostKeyNotVerifiable instead of FileNotFoundError. Explicit paths, inline host_key entries (bytes) and "none" are untouched. test_get_conn_omits_missing_default_known_hosts and the two adjusted SFTPHookAsync connection tests pin the behaviour on each side.

The five threads from @shubhamraj-git that have no reply in-thread are all answered by the code: the trigger carries no_host_key_check and SFTPOperator forwards its hook's effective value (with the TaskDeferred round-trip test), the inline known-hosts entry uses self.host or conn.host, a bare base64 host_key is normalised to ssh-rsa, the connection docs describe the opt-out as accepting unknown keys, and the async SSH tests use real Connection objects.

I re-checked the in-repo consumers against the rebased base (73a4ffa3a5): the Amazon, Google and Azure SFTP transfers, Teradata BTEQ/TPT, SFTPSensor and SSHOperator still build their hooks from the connection only, so the documented migration path applies to them. SFTPOperator.execute guarantees self.sftp_hook is set before the deferral branch reads no_host_key_check from it. The breaking-change note sits directly under the "Changelog" header in both providers, which is what the file header asks for and what the Amazon, Google and Edge changelogs do.

Two observations, neither needs a change: SSHHookAsync still has no no_host_key_check constructor argument (known_hosts="none" remains the opt-out, and the SSH changelog correctly lists only SSHHook/SFTPHook); and the PR description's "Worker hooks inherit configuration" paragraph and the 404-test count predate the rebase onto #73647, so the squash commit message may want a refresh.