Activity

1. bertrand-buffat

   added

   area:core

   kind:bugThis is a clearly a bugThis is a clearly a bug

   on Oct 30, 2021
2. boring-cyborg commented on Oct 30, 2021

   

   boring-cyborgBot

   on Oct 30, 2021

   More actions

   Thanks for opening your first issue here! Be sure to follow the issue template!
3. sweco commented on Nov 8, 2021

   

   sweco

   on Nov 8, 2021

   Last edited by sweco

   Contributor

   More actions

   Hey there, we're observing the same issue on Airflow 2.1.4 in a big DAG having hundreds of tasks. By looking in the scheduler logs from the moment when the downstream task was marked as upstream_failed it seems that it happened when a pod with a task could not be started.

   However, the question is, why is the task put in a failed state instead of retrying? And if this is normal, why does not the downstream wait for all the retries to be finished?

   Pod "pod_name" has been pending for longer than 300 seconds. It will be deleted and set to failed.
   Event: pod_name had an event of type MODIFIED
   Event: pod_name Pending
   Event: pod_name had an event of type DELETED
   Event: Failed to start pod pod_name
   Attempting to finish pod; pod_id: pod_name; state: failed; annotations: {..., 'try_number': 1}
   Changing state of task_instance, <TaskInstanceState.FAILED: 'failed'>, 'pod_name', 'airflow', '...') to failed
   Executor reports execution of task_id execution_date=execution_date exited with status failed for try_number 1
   Executor reports task instance <TaskInstance: task_id execution_date [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
   Setting task instance <TaskInstance: task_name execution_date [queued]> state to failed as reported by executor
   
   # Later on
   
   1 tasks up for execution:
   <task_id execution_date [scheduled]>
   

   Also, looking at the successful task, it indeed has two tries, from which the first one really finished at the incriminated timestamp.

   
4. sweco commented on Nov 12, 2021

   

   sweco

   on Nov 12, 2021

   Contributor

   More actions

   Steps to reproduce

   1. Prepare a relatively small cluster, e.g. 1 node with 8 GB of RAM.

   2. Deploy Airflow using Kubernetes executor there.

   3. Prepare a DAG according to the following specification

      • It tries to run 5 tasks concurrently
      • Each task specifies 2 GB of RAM in their pod requests - i.e. the sum of requested RAM is greater than the total available RAM (5 * 2 GB = 10 GB > 8 GB)
      • Each task is taking more than 300 seconds (the scheduler does not crash immediately after 300 seconds, so be sure to put some reserve, e.g. 600 seconds)

      t1 = []
      for i in range(5):
          t = BashOperator(
              task_id=f't1_{i}',
              bash_command='sleep 600 && echo Hello World!',
              executor_config={
                  'pod_override': k8s.V1Pod(
                      spec=k8s.V1PodSpec(
                          containers=[
                              k8s.V1Container(
                                  name="base",
                                  resources=k8s.V1ResourceRequirements(
                                      requests={
                                          'memory': '2000Mi',
                                          'cpu': '1000m'
                                      }
                                  )
                              )
                          ]
                      )
                  )
              }
          )
          t1.append(t)
      
      t2 = BashOperator(
          task_id='t2',
          bash_command='sleep 20 && date',
      )
      
      t1 >> t2

   4. Run the DAG, you should see that at least one of the tasks' pods will not be able to be run and will thus be in Pending state.

      

   5. After ~6 minutes the pods will be deleted by the scheduler, marked as up_for_retry and at the same time t2 will be marked as upstream_failed. The scheduler reports this error log for each task:

      ERROR - Pod "internaltestt1..." has been pending for longer than 300 seconds.It will be deleted and set to failed.
      

      In the next screenshot we can see the upstream task, whose pod was deleted, marked as up_for_retry and t2 marked as upstream_failed even though the upstream task can still succeed running on the second try.

      

   👍React with 👍2Reacted by Bertrand Buffat and Romain Fage
5. romainfage commented on Nov 15, 2021

   

   romainfage

   on Nov 15, 2021

   More actions

   Hello all!

   When looking at the issue and logs, We may have understood why the task is set to upstream_failed whereas the task is retrying. (we are using Airflow 2.1.3)

   We are getting this logs inside our scheduler:

   {kubernetes_executor.py:368} INFO - Attempting to finish pod; pod_id: POD_ID_1; state: failed; annotations: {'dag_id': 'DAG1', 'task_id': 'TASK_ID', 'execution_date': '2021-10-29T13:05:00+00:00', 'try_number': '1'}
   {kubernetes_executor.py:546} INFO - Changing state of (TaskInstanceKey(dag_id='DAG1', task_id='TASK_ID', execution_date=datetime.datetime(2021, 10, 29, 13, 5, tzinfo=tzlocal()), try_number=1), <TaskInstanceState.FAILED: 'failed'>, 'POD_ID_1', 'airflow', '164023816') to failed
   {scheduler_job.py:611} INFO - Executor reports execution of DAG1.TASK_ID execution_date=2021-10-29 13:05:00+00:00 exited with status failed for try_number 1
   {scheduler_job.py:647} ERROR - Executor reports task instance <TaskInstance: DAG1.TASK_ID 2021-10-29 13:05:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
   {scheduler_job.py:654} INFO - Setting task instance <TaskInstance: DAG1.TASK_ID 2021-10-29 13:05:00+00:00 [queued]> state to failed as reported by executor

   Our understanding is (as demonstrate @sweco), that K8S executors try to finished the pod that is not launched yet (maybe a timeout of any kind, I cannot currently pin point where and why the executors do that). That is why we see this log:

   {kubernetes_executor.py:368} INFO - Attempting to finish pod; pod_id: POD_ID_1; state: failed; annotations: {'dag_id': 'DAG1', 'task_id': 'TASK_ID', 'execution_date': '2021-10-29T13:05:00+00:00', 'try_number': '1'}
   {kubernetes_executor.py:546} INFO - Changing state of (TaskInstanceKey(dag_id='DAG1', task_id='TASK_ID', execution_date=datetime.datetime(2021, 10, 29, 13, 5, tzinfo=tzlocal()), try_number=1), <TaskInstanceState.FAILED: 'failed'>, 'POD_ID_1', 'airflow', '164023816') to failed

   Afterward, the scheduler received an event from the scheduler that change the state of the dag to failed. We can see the reception of the event here:

   {scheduler_job.py:611} INFO - Executor reports execution of DAG1.TASK_ID execution_date=2021-10-29 13:05:00+00:00 exited with status failed for try_number 1

   Afterward, the scheduler try to handle the event, to do so it look at the current state inside the metadata db of the task. However, since the pod did not start yet, the state of the task is still queued. The scheduler think because the old state is queued and the new state is failed, it is not a classic behavior so someone external should have done something. That is why we see the log:

   {scheduler_job.py:647} ERROR - Executor reports task instance <TaskInstance: DAG1.TASK_ID 2021-10-29 13:05:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?

   We can notice the question from the scheduler Was the task killed externally?, meaning that it is did not sure to handle well the case.

   When looking at the code, we can pinpoint here. Preview:

       @provide_session
       def _process_executor_events(self, session: Session = None) -> int:
               [...]
               if ti.try_number == buffer_key.try_number and ti.state == State.QUEUED:
                   Stats.incr('scheduler.tasks.killed_externally')
                   msg = (
                       "Executor reports task instance %s finished (%s) although the "
                       "task says its %s. (Info: %s) Was the task killed externally?"
                   )
                   self.log.error(msg, ti, state, ti.state, info)
   
                   request = TaskCallbackRequest(
                       full_filepath=ti.dag_model.fileloc,
                       simple_task_instance=SimpleTaskInstance(ti),
                       msg=msg % (ti, state, ti.state, info),
                   )
                   self.log.info('Setting task instance %s state to %s as reported by executor', ti, state)
                   ti.set_state(state)
                   self.processor_agent.send_callback_to_execute(request)

   By consequence, as the scheduler think something external marks the task as failed and the scheduler marks downstream tasks as upstream_failed. Then, afterward the task is retried, finished in success and is green in Airflow but no downstream tasks are launched since they are in state upstream_failed.

   I don't have currently enough insight on how the Airflow scheduler code really work, so I cannot provide a proper fix to this problem, but to fix the problem, may we do:

   • Provide another failure state to make difference between external task kill and airflow task kill?
   • Provide more information in the event received by the scheduler from the executor on the nature of the kill?
   • When the task retries, it clear the down stream task state, therefore they will not be in upstream_failed state?

   Thank you for reading me

   👍React with 👍4Reacted by Bertrand Buffat, Mikhail Bolshakov, Daniel Birnstiel and Damotorie
6. eladkal commented on Nov 24, 2021

   

   eladkal

   on Nov 24, 2021

   Contributor

   More actions

   cc @jedcunningham
7. eladkal

   added

   provider:cncf-kubernetesKubernetes (k8s) provider related issuesKubernetes (k8s) provider related issues

   _eol_affected_version:2.1Issues Reported for 2.1Issues Reported for 2.1

   area:scheduler

   on Nov 24, 2021
8. kehao95 commented on Jan 18, 2022

   

   kehao95

   on Jan 18, 2022

   Last edited by kehao95

   More actions

   We are experiencing the same issue at v2.1.4. task is showing upstream_failed
   It may be related to resources issue. We have 2 env dev&prod. The Dev has very limited resources which this happened frequently. And Production has much larger CPU/Mem quotas where the issue happened less frequently.

   ---

   Update: experiencing the same issue at v2.1.2 after downgrading.
9. aprettyloner commented on Jan 20, 2022

   

   aprettyloner

   on Jan 20, 2022

   Last edited by aprettyloner

   More actions

   Hi @jedcunningham @eladkal has this bug been fixed in 2.2.1+?

   version: 2.1.4
   gitSync: enabled

   We are seeing this intermittently every day with both complex and simple dags.

   More complex dags

   

   

   Simple linear dags

   

   
10. aprettyloner commented on Jan 20, 2022

    

    aprettyloner

    on Jan 20, 2022

    More actions

    Hi @jedcunningham @eladkal has this bug been fixed in 2.2.1+?

    version: 2.1.4 gitSync: enabled

    We are seeing this intermittently every day with both complex and simple dags.

    More complex dags

    

    

    Simple linear dags

    

    

    cc @ephraimbuddy
11. eladkal commented on Oct 6, 2022

    

    eladkal

    on Oct 6, 2022

    Contributor

    More actions

    @aprettyloner does the issue still happen in latest airflow version + latest provider?
12. bertrand-buffat commented on Oct 6, 2022

    

    bertrand-buffat

    on Oct 6, 2022

    Last edited by bertrand-buffat

    Author

    More actions

    In our case, we implemented a dag that monitored the problem every 10 minutes and cleared the blocked tasks if any. Then the error ceased to appear after an upgrade from aws.
    I can close the issue if you want.
13. Birne94 commented on Oct 6, 2022

    

    Birne94

    on Oct 6, 2022

    More actions

    In our case, we implemented a dag that monitored the problem every 10 minutes and cleared the blocked tasks if any. Then the error ceased to appear after an upgrade from aws.

    @bertrand-buffat would you be able to share the code for that DAG and how you are finding/clearing the affected tasks? I think this would be very helpful for us as well.
14. bertrand-buffat commented on Oct 6, 2022

    

    bertrand-buffat

    on Oct 6, 2022

    Last edited by bertrand-buffat

    Author

    More actions

    Sure.
    Operator:

    class AirflowBandAids(BaseOperator):
        """Base Operator to help perform fixes to bypass known airflow bugs"""
    
        @provide_session
        def clear_tasks(
            self,
            dag_id: str,
            run_id: str,
            task_ids: List[str],
            session=None,
        ):
            """Fetch and create task instance object that should be cleared and then clear them"""
            tis = session.query(TaskInstance).filter(
                TaskInstance.dag_id == dag_id,
                TaskInstance.run_id == run_id,
                TaskInstance.task_id.in_(task_ids),
            )
            all_tis = tis.all()
            self.log.info(f"Clear task {[ti.task_id for ti in all_tis]} for dag {dag_id} with this run id {run_id}")
            clear_task_instances(tis=all_tis, session=session)
    
        def get_task_per_dag_run(
            self,
            query_returning_dag_task_run_id,
            airflow_conn_id: str,
        ) -> Dict[Tuple[str, str], TaskInstance]:
            """Returns the list of task per dag_run"""
            hook = PostgresHook(airflow_conn_id)
    
            self.log.info(f"Execute Query: {query_returning_dag_task_run_id}")
            tasks_to_clear = hook.get_records(query_returning_dag_task_run_id)
            tasks_by_dags = defaultdict(list)
            for dag_id, run_id, task_id in tasks_to_clear:
                tasks_by_dags[dag_id, run_id].append(task_id)
            return dict(tasks_by_dags)
    
    class ClearFalseUpstreamFailedTaskOperator(AirflowBandAids):
        """Clears tasks that cab appear with a state upstream failed even though there are no task failed upstream"""
    
        query_template = """SELECT
      upstream_failed.dag_id,
      upstream_failed.run_id,
      upstream_failed.task_id
    FROM (
     SELECT dag_id, task_id, state, run_id
     FROM task_instance
     WHERE state = 'upstream_failed'
     AND start_date BETWEEN '{start_date}' and '{end_date}'
    ) AS upstream_failed
    LEFT JOIN task_instance t
      ON t.dag_id = upstream_failed.dag_id
      AND t.run_id = upstream_failed.run_id
      AND t.state = 'failed'
    WHERE t.state is null
    """
    
        def __init__(
            self,
            airflow_conn_id,
            hour_delta,
            *args,
            **kwargs,
        ):
            """
            Args:
                airflow_conn_id: The connection id of airflow metadata backend
                hour_delta: The delta to apply to these task instances execution date.
                            The date is used as range to look for false upstream_failed
                            task in the metadata DB.
            """
            super().__init__(*args, **kwargs)
            self.airflow_conn_id = airflow_conn_id
            self.hour_delta = hour_delta
    
        def execute(self, context):
            tasks_by_dags = self.get_task_per_dag_run(
                self.query_template.format(
                    start_date=context["logical_date"].add(hours=-self.hour_delta),
                    end_date=context["data_interval_end"],
                ),
                airflow_conn_id=self.airflow_conn_id,
            )
    
            if not tasks_by_dags:
                raise AirflowSkipException("No task to clear")
    
            for (dag_id, run_id), task_ids in tasks_by_dags.items():
                self.clear_tasks(dag_id, run_id, task_ids)
    

    dag:

    from doctolib_plugins.operators.airflow_metadata_operator import ClearFalseUpstreamFailedTaskOperator
    
    with DAG(
        dag_id="wkf_clear_airflow_tasks",
        schedule_interval="*/10 * * * *",
        default_args=obtain_default_args(),
        max_active_runs=1,
        catchup=False,
    ) as dag:
        ClearFalseUpstreamFailedTaskOperator(
            task_id="clear_false_upstream_failed_tasks",
            airflow_conn_id=airflow_backend_db_conn_id,
            hour_delta=3,
        )
    

    👍React with 👍1Reacted by Daniel Birnstiel

    🎉React with 🎉1Reacted by Daniel Birnstiel
15. aprettyloner commented on Oct 6, 2022

    

    aprettyloner

    on Oct 6, 2022

    More actions

    @aprettyloner does the issue still happen in latest airflow version + latest provider?

    @eladkal - Issue is not occurring on updated version:

    Version: [v2.2.3](https://pypi.python.org/pypi/apache-airflow/2.2.3)
    Git Version: .release:2.2.3+06c82e17e9d7ff1bf261357e84c6013ccdb3c241
    
16. eladkal

    closed this as completedon Oct 6, 2022
17. eladkal

    added

    Can't ReproduceThe problem cannot be reproducedThe problem cannot be reproduced

    on Oct 6, 2022