Skip to content

dag_run.conf is NoneType when running example_trigger_target_dag from example_dags  #9852

New issue
New issue
Closed
#9853
Closed
dag_run.conf is NoneType when running example_trigger_target_dag from example_dags #9852
#9853
Assignees
kaxil
Labels
kind:bugThis is a clearly a bug
Milestone
Airflow 1.10.12

Description

@uns1
uns1
opened on Jul 16, 2020

Apache Airflow version: 1.10.9 & 1.10.11
Environment:

  • Cloud provider or hardware configuration: Local PC running Docker on Windows through WSL2
  • OS (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
  • Kernel (e.g. uname -a): Linux ba34d8b76277 4.19.104-microsoft-standard Improving the search functionality in the graph view #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64 GNU/Linux
  • Install tools: Docker + Python 3.7
  • Others: PostgreSQL + Local Executor

What happened:
I'm trying to run the example dag that triggers another dag. When i run the example, the run is unsuccessful because dag_run.conf is NoneType rather than the conf passed in the controller dag.

Moreover, the dag runs fine if run from the CLI

What you expected to happen:
The conf from the trigger dag should not be NoneType

What do you think went wrong?
Unsure - previous JIRA post suggests this has something to do with pickling. However, that post refers to SQLite being the root of the issue whereas i'm using Postgres

How to reproduce it:
Running the example dag for triggering other dags on the above configuration - I suspect this is not a common issue as I do not find a lot of resources related to this error on git, jira, stackoverflow etc

Anything else we need to know:
Exact replica of https://issues.apache.org/jira/browse/AIRFLOW-46 However, the JIRA post suggests that this only happens on SQLite whereas this is happening on Postgres for me

How often does this happen?
Problem occurs every time I run the example_dag for triggering another dag

Any relevant logs to include?

DAG Run Log 
[2020-07-16 01:12:15,968] {{taskinstance.py:670}} INFO - Dependencies all met for <TaskInstance: example_trigger_target_dag.run_this 2020-07-16T01:12:12.248293+00:00 [queued]>

[2020-07-16 01:12:16,018] {{taskinstance.py:670}} INFO - Dependencies all met for <TaskInstance: example_trigger_target_dag.run_this 2020-07-16T01:12:12.248293+00:00 [queued]>

[2020-07-16 01:12:16,018] {{taskinstance.py:880}} INFO - 
--------------------------------------------------------------------------------
[2020-07-16 01:12:16,018] {{taskinstance.py:881}} INFO - Starting attempt 1 of 1
[2020-07-16 01:12:16,018] {{taskinstance.py:882}} INFO - 
--------------------------------------------------------------------------------
[2020-07-16 01:12:16,071] {{taskinstance.py:901}} INFO - Executing <Task(PythonOperator): run_this> on 2020-07-16T01:12:12.248293+00:00

[2020-07-16 01:12:16,073] {{standard_task_runner.py:54}} INFO - Started process 575 to run task

[2020-07-16 01:12:16,093] {{standard_task_runner.py:77}} INFO - Running: ['airflow', 'run', 'example_trigger_target_dag', 'run_this', '2020-07-16T01:12:12.248293+00:00', '--job_id', '25', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/test_dag_2.py', '--cfg_path', '/tmp/tmp0tn7w7v2']

[2020-07-16 01:12:16,093] {{standard_task_runner.py:78}} INFO - Job 25: Subtask run_this

[2020-07-16 01:12:16,167] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: example_trigger_target_dag.run_this 2020-07-16T01:12:12.248293+00:00 [running]> 4b821a8e9fcc

[2020-07-16 01:12:16,186] {{logging_mixin.py:112}} INFO - <class 'airflow.models.dagrun.DagRun'>

[2020-07-16 01:12:16,186] {{taskinstance.py:1150}} ERROR - 'NoneType' object is not subscriptable
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 113, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/airflow/dags/test_dag_2.py", line 44, in run_this_func
    print("Remotely received value of {} for key=message".format(context["dag_run"].conf["message"]))
TypeError: 'NoneType' object is not subscriptable

[2020-07-16 01:12:16,191] {{taskinstance.py:1194}} INFO - Marking task as FAILED. dag_id=example_trigger_target_dag, task_id=run_this, execution_date=20200716T011212, start_date=20200716T011215, end_date=20200716T011216

[2020-07-16 01:12:20,926] {{local_task_job.py:102}} INFO - Task exited with return code 1

Activity

  1. uns1
    added
    kind:bugThis is a clearly a bug
    on Jul 16, 2020

  2. boring-cyborg commented on Jul 16, 2020

    @boring-cyborg
    boring-cyborgBot
    on Jul 16, 2020

    Thanks for opening your first issue here! Be sure to follow the issue template!

  3. kaxil
    self-assigned this
    on Jul 16, 2020
  4. kaxil
    added this to the Airflow 1.10.12 milestone on Jul 16, 2020
  5. kaxil
    closed this as completedin #9853on Jul 16, 2020
  6. kaxil
    added a commit that references this issue on Jul 16, 2020

    Fix DagRun.conf when using trigger_dag API (#9853)

    31cab8f
  7. manesioz
    added a commit that references this issue on Jul 16, 2020

    sync fork from main (#1)

    3641615

  8. uns1 commented on Jul 17, 2020

    @uns1
    uns1
    on Jul 17, 2020
    Author

    Thanks @kaxil! While this ensures that conf will always return a dictionary and not None, my issue is that conf is not being passed from the controller to the target dag at all - as per the example_dags.

    In the context of the example, the key value pair of "message":"Hello World" is not making it through to the target dag.

  9. uns1 commented on Jul 17, 2020

    @uns1
    uns1
    on Jul 17, 2020
    Author

    Thanks @kaxil! While this ensures that conf will always return a dictionary and not None, my issue is that conf is not being passed from the controller to the target dag at all - as per the example_dags.

    In the context of the example, the key value pair of "message":"Hello World" is not making it through to the target dag.

    I have opened this as a separate issue here : #9870

  10. kaxil commented on Jul 17, 2020

    @kaxil
    kaxil
    on Jul 17, 2020
    Member

    Thanks @kaxil! While this ensures that conf will always return a dictionary and not None, my issue is that conf is not being passed from the controller to the target dag at all - as per the example_dags.
    In the context of the example, the key value pair of "message":"Hello World" is not making it through to the target dag.

    I have opened this as a separate issue here : #9870

    oh I didn't realize, can you share your DAG in the new issue you created for me to reproduce the error

  11. kaxil commented on Jul 17, 2020

    @kaxil
    kaxil
    on Jul 17, 2020
    Member

    Saw your comment here: #9853 (comment) will try to reproduce it today

  12. leahecole
    added 2 commits that reference this issue on Sep 15, 2021

    Fix DagRun.conf when using trigger_dag API (#9853)

    f7119ec

    Fix DagRun.conf when using trigger_dag API (#9853)

    01b3cef

  13. 9 remaining items

  14. leahecole
    added a commit that references this issue on Dec 7, 2022

    Fix DagRun.conf when using trigger_dag API (#9853)

    79aeeee
  15. leahecole
    added a commit that references this issue on Jan 27, 2023

    Fix DagRun.conf when using trigger_dag API (#9853)

    a7694ef
  16. kosteev
    added 3 commits that reference this issue on Sep 11, 2024

    Fix DagRun.conf when using trigger_dag API (#9853)

    5dcc86c

    Fix DagRun.conf when using trigger_dag API (#9853)

    55f567c

    Fix DagRun.conf when using trigger_dag API (#9853)

    0f58637
  17. kosteev
    added a commit that references this issue on Nov 6, 2024

    Fix DagRun.conf when using trigger_dag API (#9853)

    850c21b
  18. kosteev
    added a commit that references this issue on May 1, 2025

    Fix DagRun.conf when using trigger_dag API (#9853)

    363d0a9
  19. kosteev
    added a commit that references this issue on May 21, 2025

    Fix DagRun.conf when using trigger_dag API (#9853)

    f27d1cf
  20. kosteev
    added a commit that references this issue on Sep 16, 2025

    Fix DagRun.conf when using trigger_dag API (#9853)

    451b94f
  21. kosteev
    added a commit that references this issue on Oct 15, 2025

    Fix DagRun.conf when using trigger_dag API (#9853)

    f506a85
  22. kosteev
    added a commit that references this issue on Feb 20, 2026

    Fix DagRun.conf when using trigger_dag API (#9853)

    fdabfc2
  23. kosteev
    added a commit that references this issue on Apr 18, 2026

    Fix DagRun.conf when using trigger_dag API (#9853)

    e4f40ca
  24. kosteev
    added 2 commits that reference this issue on Jul 22, 2026

    Fix DagRun.conf when using trigger_dag API (#9853)

    9312d61

    Fix DagRun.conf when using trigger_dag API (#9853)

    3ccad6f
  25. kosteev
    added a commit that references this issue on Aug 29, 2026

    Fix DagRun.conf when using trigger_dag API (#9853)

    ecc6bd3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind:bugThis is a clearly a bug

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions