AWS Glue Hook to implement enhanced logging #23832

Activity

1. ak-arun

   added

   kind:featureFeature RequestsFeature Requests

   on May 20, 2022
2. ak-arun

   changed the title [-]AWS Glue Hook to implement more verbos logging[/-] [+]AWS Glue Hook to implement enhanced logging[/+] on May 20, 2022
3. potiuk commented on May 22, 2022

   

   potiuk

   on May 22, 2022

   Member

   More actions

   @ak-arun - maybe you would like to add Pull Request fixing it ? Seems ike an easy thing to do that would not require deep knowledge of Airlow itself and you could test it locally with your setup ? Can I assign it to you? This is the easiest and fastest way to get it implemented :). Otherwise I mark it as "good-first-issue".
4. potiuk

   added

   good first issue

   on May 22, 2022
5. uranusjr

   added

   provider:amazonAWS/Amazon - related issuesAWS/Amazon - related issues

   area:providers

   on May 23, 2022
6. Dark-Knight11 commented on May 25, 2022

   

   Dark-Knight11

   on May 25, 2022

   Last edited by Dark-Knight11

   Contributor

   More actions

   Hi, I would like to work on this feature. Can you please guide me a little on how to proceed with it?

   I looked into glue.py file and I think I have to make changes over here.
   How can I add the logs from cloudwatch/glue over here?
   Do I need to use AwsLogsHook?

   airflow/airflow/providers/amazon/aws/hooks/glue.py

   Lines 161 to 164 in 1d53bec

   	else:
   	self.log.info(
   	"Polling for AWS Glue Job %s current run state with status %s", job_name, job_run_state
   	)
7. Taragolis commented on May 25, 2022

   

   Taragolis

   on May 25, 2022

   Contributor

   More actions

   @Dark-Knight11 I would like to warn about side effect of fetch AWS Glue Logs into Airflow Tasks logs (already mentioned in the almost the same issue #23900 (comment)).

   If your Glue Job uses 40 DPU that mean it spawn minimum 1 driver and 39 workers/executors and all of them will create errors and output logs, as result

   1. You need to find all of CloudWatch Logs Prefix started with job_id in correct log group, by default Glue uses
      • /aws-glue/jobs/output - for output logs
      • /aws-glue/jobs/error - for error logs
      • /aws-glue/jobs/logs-v2 - (optional) for continuous logging
   2. You need to be sure that you fetch from all prefixes and this prefixes doesn't created in the same time
   3. Be sure that fetch logging processes/threads do not use all CPU/Memory/IO of Airflow Worker
8. Dark-Knight11 commented on May 25, 2022

   

   Dark-Knight11

   on May 25, 2022

   Contributor

   More actions

   Ok, Thanks. I'll look into it
9. josh-fell

   assigned

   Dark-Knight11

   on Jun 3, 2022
10. ferruzzi commented on Jul 19, 2022

    

    ferruzzi

    on Jul 19, 2022

    Contributor

    More actions

    Closed by #25142

    👍React with 👍1Reacted by Arun A K
11. eladkal

    closed this as completedon Jul 19, 2022
12. ak-arun commented on Oct 11, 2022

    

    ak-arun

    on Oct 11, 2022 via email

    Author

    More actions

    Hi Andrey, Point taken. I am thinking if we just publish the logs when the job state changes to Error/Failed. Also, instead of mining all executor logs + the driver, I think we can read the 2 recently added job insights streams - https://docs.aws.amazon.com/glue/latest/dg/monitor-job-insights.html For sure, this is a new feature in Glue and older jobs may not have these streams enabled. We can mention it works only if job-insights are enabled?-Thoughts ?

    …

    -- Thanks & Regards *Arun A K*

    On Wed, May 25, 2022 at 1:10 PM Andrey Anshin ***@***.***> wrote: @Dark-Knight11 <https://github.com/Dark-Knight11> I would like to warn about side effect of fetch AWS Glue Logs into Airflow Tasks logs (already mentioned in the almost the same issue #23900 (comment) <#23900 (comment)>). If your Glue Job uses 40 DPU that mean it spawn minimum 1 driver and 39 workers/executors and all of them will create errors and output logs, as result 1. You need to find all of CloudWatch Logs Prefix started with job_id in correct log group, by default Glue uses - /aws-glue/jobs/output - for output logs - /aws-glue/jobs/error - for error logs - /aws-glue/jobs/logs-v2 - (optional) for continuous logging 2. You need to be sure that you fetch from all prefixes and this prefixes doesn't created in the same time 3. Be sure that fetch logging processes/threads do not use all CPU/Memory/IO of Airflow Worker — Reply to this email directly, view it on GitHub <#23832 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVRXFOQEOW4BUUI2YU5FHLVLZNHPANCNFSM5WP75PDA> . You are receiving this because you were mentioned.Message ID: ***@***.***>