Skip to content

Speed up secret masking of task log lines - #75232

Open
olmaks wants to merge 1 commit into
apache:mainfrom
olmaks:speed-up-log-secret-masking
Open

olmaks wants to merge 1 commit into
apache:mainfrom
olmaks:speed-up-log-secret-masking

Conversation

@olmaks

@olmaks olmaks commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Every task log line goes through SecretsMasker.should_hide_value_for_key, once per key (dag_id, ti_id, logger, ...), and #74840 made it heavier:

def should_hide_value_for_key(self, name):
    if isinstance(name, str) and self.hide_sensitive_var_conn_fields:
        name = name.strip()
        normalized = _NON_WORD_RE.sub("_", name.lower())
        split = _CAMEL_ACRONYM_RE.sub(r"\1_\2", _CAMEL_LOWER_UPPER_RE.sub(r"\1_\2", name))
        split = _NON_WORD_RE.sub("_", split.lower())
        return any(s in normalized or s in split for s in self.sensitive_variables_fields)

The answer only depends on the name and the field contents, so it moves into a module-level lru_cache keyed by (name, tuple(sensitive_variables_fields)) — reassigning or mutating the fields can't return a stale answer, and the cache is bounded because variable and connection-extra keys come from user data. redact_jwt ran its regex on every string value; strings without eyJ now skip it.

In this logging-heavy workload, processing logs in the supervisor creates backpressure on the task, so reducing masking overhead reduces task duration. LocalExecutor, a task writing 100k lines, Linux x86_64; image apache/airflow:3.4.0b1 with these two files replaced by the versions from main and from this branch. Median of 5 fresh containers per side (each the mean of 2 runs), the two sides run back to back in randomised order:

before after
BashOperator 12.5 s 6.7 s
logging.info 12.6 s 6.7 s
print 5.8 s 3.3 s

Each of the 5 back-to-back pairs gave a ratio of 0.50–0.54 (Bash), 0.53–0.55 (logging.info) and 0.52–0.60 (print).

Tests. Two new tests cover the cache and the JWT shortcut; both fail if the source change is reverted. Comparing old and new code on 120k key lookups (including in-place edits of the field list) and 100k log events gave identical masking results. An unusual compatibility edge remains: an unhashable str subclass used as a name, or an unhashable sensitive-field entry, now raises TypeError during cache lookup. redact catches this and returns <redaction-failed>. Airflow's configuration supplies ordinary strings.

Local: shared/secrets_masker 174 passed (1 xfailed), shared/logging 111 passed; test_http_access_log, test_decorators and the OpenLineage redactor tests pass; prek clean. In the four task-sdk modules I ran (test_log, test_operator, test_supervisor, test_task_runner), 1075 pass and 11 fail (secrets-backend count and TriggerDagRun call assertions); the same 11 failures also occur on the base commit.

related: #71721


Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Claude Code (Opus 5.5) following the guidelines
Used to profile the task log path, write the change and its tests. I reviewed every line and re-ran the benchmarks and tests myself.


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Every task log line passes through the secrets masker and the JWT
redactor in the supervisor, and the JWT redactor also runs in the task
process. The masker re-normalised every key of every line with several
regex substitutions and compared it against all sensitive field names,
even though the answer only depends on the key name and the configured
fields. The JWT redactor ran its regex on every string value, including
the vast majority that cannot contain a token.

With a high log volume the supervisor becomes the bottleneck and the
task blocks while writing its logs, so this cost shows up directly in
task duration.
@olmaks
olmaks marked this pull request as ready for review October 20, 2026 22:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant