Webserver doesn't mask rendered fields for pending tasks #22738

Activity

1. m1racoli

   added

   area:core

   kind:bugThis is a clearly a bugThis is a clearly a bug

   on Apr 4, 2022
2. tirkarthi commented on Apr 5, 2022

   

   tirkarthi

   on Apr 5, 2022

   Contributor

   More actions

   I have created #22754 as a fix. Can you please confirm if MASK_SECRETS_IN_LOGS is enabled in your settings?
3. m1racoli commented on Apr 5, 2022

   

   m1racoli

   on Apr 5, 2022

   ContributorAuthor

   More actions

   I have created #22754 as a fix. Can you please confirm if MASK_SECRETS_IN_LOGS is enabled in your settings?

   I am not sure that variable is intended to be user configurable. But HIDE_SENSITIVE_VAR_CONN_FIELDS is set to True as per Airflow config reference.
4. tirkarthi commented on Apr 5, 2022

   

   tirkarthi

   on Apr 5, 2022

   Contributor

   More actions

   Thanks, MASK_SECRETS_IN_LOGS should be set for masking of secret to work. It is set automatically when airflow tasks run is called. The sequential scheduler I use for local testing internally calls the airflow tasks run command to run tasks and thus I was confused over how masking is done when task is executed though the taskinstance.py file had not set the variable. I left a note in the PR over whether we should enable this just during rendering the task template in UI.

   #22754 (comment)

   👍React with 👍1Reacted by Cedrik Neumann
5. ephraimbuddy

   closed this as completedin #22754on May 20, 2022
6. dstandish

   added a commit that references this issue on May 21, 2022

   Mask sensitive values for not-yet-running TIs (#23807)

   ...

   This issue will close once commit 2dc8063 is merged into the 'main' branch.

   2dc8063
7. ephraimbuddy

   added a commit that references this issue on May 27, 2022

   Mask sensitive values for not-yet-running TIs (#23807)

   ...

   This issue will close once commit 09be0c5 is merged into the 'main' branch.

   09be0c5
8. andrewdanks

   added a commit that references this issue on Jun 2, 2022

   pull master (#25)

   ...

   This issue will close once commit b10afd1 is merged into the 'master' branch.

   b10afd1
9. hugochinchilla commented on Oct 20, 2022

   

   hugochinchilla

   on Oct 20, 2022

   More actions

   I'm still seing unmasked values for secrets on the rendered section for a pending task, once it runs the values get masked. In my case this is happening in the environment field of a docker operator. I don't know if the problem happens only because it's a different field, maybe the tests for this should be extended.

   I'm using Airflow 2.3.4
10. hugochinchilla commented on Oct 20, 2022

    

    hugochinchilla

    on Oct 20, 2022

    More actions

    Here is an example on how the issue can still be reproduced:

    from datetime import timedelta
    
    from airflow import DAG
    from airflow.providers.docker.operators.docker import DockerOperator
    from airflow.sensors.time_delta import TimeDeltaSensor
    from airflow.utils.dates import days_ago
    from airflow.utils.log.secrets_masker import mask_secret
    
    secret_value = "you cannot see this"
    
    mask_secret(secret_value)
    
    with DAG(
        "secrets",
        start_date=days_ago(1),
        schedule_interval=None,
    ) as dag:
        wait = TimeDeltaSensor(
            task_id="wait",
            delta=timedelta(seconds=30),
        )
    
        task = wait >>     DockerOperator(
            task_id="docker",
            environment={
                "MY_VARIABLE": secret_value,
            },
            image="hello-world",
        )
11. m1racoli commented on Oct 21, 2022

    

    m1racoli

    on Oct 21, 2022

    Last edited by m1racoli

    ContributorAuthor

    More actions

    @hugochinchilla I believe you need to store your secret in an Airflow variable or connection and then use jinja to render it in your MY_VARIABLE value in order to take advantage of secrets masking. Not sure this works for hard-coded values.

    Also secrets should not be hard-coded or loaded during DAG parsing (more in Airflow Best Practices).
12. hugochinchilla commented on Oct 25, 2022

    

    hugochinchilla

    on Oct 25, 2022

    More actions

    Actually this is a simplification, the problem I'm having is actually using data inside a Connection, but that connection is serialized as base64 in a factory that creates the operators, the code calls mask_secret() but the value is exposed until the task actually runs.

    I've mitigated it creating a macro to call mask_secret() and return the value, so I can use something like {{ macros.plugin_name.mask_secret(some_secret) }} and the value is masked even when the task is pending.