Skip to content

airflow scheduler ERROR - Exception when executing SchedulerJob._run_scheduler_loop #23682

Description

@rafficghani

Apache Airflow version

2.2.2

What happened

The scheduler goes down

[2022-05-12 11:14:06,586] {scheduler_job.py:1114} INFO - Resetting orphaned tasks for active dag runs
[2022-05-12 11:14:06,806] {scheduler_job.py:644} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/airflow/airflowvirt/lib64/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 628, in _execute
self._run_scheduler_loop()
File "/airflow/airflowvirt/lib64/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 721, in _run_scheduler_loop
next_event = timers.run(blocking=False)
File "/usr/lib64/python3.6/sched.py", line 154, in run
action(*argument, **kwargs)
File "/airflow/airflowvirt/lib64/python3.6/site-packages/airflow/utils/event_scheduler.py", line 36, in repeat
action(*args, **kwargs)
File "/airflow/airflowvirt/lib64/python3.6/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/airflow/airflowvirt/lib64/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1117, in adopt_or_reset_orphaned_tasks
for attempt in run_with_db_retries(logger=self.log):
File "/airflow/airflowvirt/lib64/python3.6/site-packages/tenacity/init.py", line 382, in iter
do = self.iter(retry_state=retry_state)
File "/airflow/airflowvirt/lib64/python3.6/site-packages/tenacity/init.py", line 349, in iter
return fut.result()
File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/airflow/airflowvirt/lib64/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1162, in adopt_or_reset_orphaned_tasks
to_reset = self.executor.try_adopt_task_instances(tis_to_reset_or_adopt)
File "/airflow/airflowvirt/lib64/python3.6/site-packages/airflow/executors/celery_executor.py", line 485, in try_adopt_task_instances
self.adopted_task_timeouts[ti.key] = ti.queued_dttm + self.task_adoption_timeout
TypeError: unsupported operand type(s) for +: 'NoneType' and 'datetime.timedelta'
[2022-05-12 11:14:08,027] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 3929915

What you think should happen instead

No response

How to reproduce

No response

Operating System

Redhat x86_64 GNU/Linux

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions