Skip to content

CloudWatch CreateLogGroup error on already existing LogGroup #18549

Closed
Closed
CloudWatch CreateLogGroup error on already existing LogGroup#18549
Labels
area:coregood first issuekind:bugThis is a clearly a bug

Description

@jo-gre
jo-gre
opened on Sep 27, 2021

Apache Airflow version

2.1.2

Operating System

Ubuntu Docker

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==2.0.0
apache-airflow-providers-celery==2.0.0
apache-airflow-providers-cncf-kubernetes==2.0.0
apache-airflow-providers-docker==2.0.0
apache-airflow-providers-elasticsearch==2.0.2
apache-airflow-providers-ftp==2.0.0
apache-airflow-providers-google==4.0.0
apache-airflow-providers-grpc==2.0.0
apache-airflow-providers-hashicorp==2.0.0
apache-airflow-providers-http==2.0.0
apache-airflow-providers-imap==2.0.0
apache-airflow-providers-microsoft-azure==3.0.0
apache-airflow-providers-mysql==2.0.0
apache-airflow-providers-neo4j==2.0.0
apache-airflow-providers-postgres==2.0.0
apache-airflow-providers-redis==2.0.0
apache-airflow-providers-sendgrid==2.0.0
apache-airflow-providers-sftp==2.0.0
apache-airflow-providers-slack==4.0.0
apache-airflow-providers-sqlite==2.0.0
apache-airflow-providers-ssh==2.0.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

    config:
      core:
        executor: 'CeleryExecutor'
      logging:
        remote_logging: 'True'
        remote_base_log_folder: 'cloudwatch://arn:aws:logs:eu-central-1:<redacted>:log-group:airflow'
        remote_log_conn_id: 'cloudwatch'

The log group is already created in Terraform.

What happened


[2021-09-27 15:03:58,541: ERROR/ForkPoolWorker-15] Failed to execute task An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts::<redacted>:assumed-role/airflow-service-account-role/botocore-session-1632755037 is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:eu-central-1:<redacted>:log-group:airflow:log-stream:.
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 117, in _execute_in_fork
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 228, in task_run
    ti.init_run_context(raw=args.raw)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2031, in init_run_context
    self._set_context(self)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 54, in _set_context
    set_context(self.log, context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 173, in set_context
    handler.set_context(value)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py", line 81, in set_context
    self.handler = watchtower.CloudWatchLogHandler(
  File "/home/airflow/.local/lib/python3.8/site-packages/watchtower/__init__.py", line 148, in __init__
    _idempotent_create(self.cwl_client, "create_log_group", logGroupName=self.log_group)
  File "/home/airflow/.local/lib/python3.8/site-packages/watchtower/__init__.py", line 15, in _idempotent_create
    method_callable(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts:<redacted>}:assumed-role/airflow-service-account-role/botocore-session-1632755037 is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:eu-central-1:<redacted>:log-group:airflow:log-stream:

What you expected to happen

I would expect Airflow to create a stream and write the logs, not to create a log-group inside a log-group.

How to reproduce

Run any dag with the chart config.

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. jo-gre
    added
    area:core
    kind:bugThis is a clearly a bug
    on Sep 27, 2021

  2. boring-cyborg commented on Sep 27, 2021

    @boring-cyborg
    boring-cyborgBot
    on Sep 27, 2021

    Thanks for opening your first issue here! Be sure to follow the issue template!

  3. potiuk
    added
    good first issue
    hacktoberfest
    on Sep 29, 2021

  4. anaynayak commented on Oct 7, 2021

    @anaynayak
    anaynayak
    on Oct 7, 2021
    Contributor

    @potiuk I think I can try this one.

    My current understanding of the code + changes required:

    1. CloudwatchTaskHandler delegates to watchtower.CloudWatchLogHandler which provides the option to create a log group if it doesn't exist (true by default) https://github.com/kislyuk/watchtower/blob/develop/watchtower/__init__.py#L84
    2. Do we intend to continue with the existing behavior but allow users to override based on config values defined ? e.g. conf.get('logging', 'CLOUDWATCH_CREATE_LOG_GROUP', fallback=True) similar to https://github.com/apache/airflow/blob/main/airflow/config_templates/airflow_local_settings.py#L209 ?

    Let me know if thats the approach. Can do the changes accordingly.

  5. potiuk commented on Oct 10, 2021

    @potiuk
    potiuk
    on Oct 10, 2021
    Member

    Hard to say. I am not expert on cloud-watch - but maybe others can help :)

  6. anaynayak commented on Oct 12, 2021

    @anaynayak
    anaynayak
    on Oct 12, 2021
    Contributor

    Was curious more from a config standpoint. I believe the way to specify/override the default config is from airflow_local_settings.py. Will create a PR. Will make it easier to iterate on it.

    If anyone else gets to reviewing the approach, please let me know :)

  7. o-nikolas commented on Oct 28, 2021

    @o-nikolas
    o-nikolas
    on Oct 28, 2021
    Contributor

    Hey folks,

    watchtower does an idempotent create for the log group as you can see here.. If the log group already exists it will catch that error and ignore it silently.

    The issue you actually hit is an authentication issue, as seen in your exception message:

    botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts:}:assumed-role/airflow-service-account-role/botocore-session-1632755037 is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:eu-central-1::log-group:airflow:log-stream:

    The role you're using doesn't have permission for the log group creation. It's worth double checking that role has all the required permissions for cloudwatch logging (creating groups, streams and uploading records).

  8. o-nikolas commented on Nov 3, 2021

    @o-nikolas
    o-nikolas
    on Nov 3, 2021
    Contributor

    Given the discussion in 19022, I think this issue can be resolved/closed?

  9. jo-gre commented on Nov 3, 2021

    @jo-gre
    jo-gre
    on Nov 3, 2021
    Author

    Hey folks,

    sorry that I didn't respond for that long.

    I also figured out that this looks like an authentication problem but even with all privileges given to airflow the error still occurred. I was wondering if airflow maybe tries to create the log group inside the log stream. The error arn:aws:logs:eu-central-1:<redacted>:log-group:airflow:log-stream: looks like the correct path, but I don't see a reason why AWS would require the sub-resouce :log-stream: when calling to create a log-group.

  10. o-nikolas commented on Nov 3, 2021

    @o-nikolas
    o-nikolas
    on Nov 3, 2021
    Contributor

    What I'm quite sure of is that the issue is constrained to your configuration. Otherwise we'd have this exception for all users with logging to cloudwatch.

    I don't think the boto3 api would return an authentication issue if the log group name was formatted incorrectly. The airflow-service-account-role you're using likely is still missing the correct policies.
    As for the strangely formatted loggroup name, check the config you're using to specify it. You can even paste that here to get a second pair of eyes on it.

  11. eladkal
    removed
    hacktoberfest
    on Feb 1, 2022

  12. ptrhck commented on May 17, 2022

    @ptrhck
    ptrhck
    on May 17, 2022

    I am facing the very same issue on Airflow 2.2.2

    Failed to execute task An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts::XXXXXXX:assumed-role/airflow-ecs-task-role-staging/airflow_logs_cloudwatch is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:eu-central-1:XXXXXXX:assumed:log-group:airflow-v2-staging-dags:log-stream: because no identity-based policy allows the logs:CreateLogGroup action.

    The Cloudwatch log group is already created using Cloudformation and the policies accordingly:

    Resources:

  DagsLogs:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub airflow-v2-${Stage}-dags

  TaskRole:
    Type: AWS::IAM::Role
    Properties:
      Policies:
        - PolicyName: !Sub ${EnvironmentName}-logs-dags-${Stage}
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource:
                  - !GetAtt DagsLogs.Arn

    The Airflow cofiguration variables are the following:

    Environment:
  - Name: AIRFLOW_CONN_LOGS_CLOUDWATCH
    Value: !Sub 'cloudwatch://cloudwatch?aws_account_id=XXXXXXX&role_arn=arn%3Aaws%3Aiam%3A%3A919107267526%3Arole%2Fairflow-ecs-task-role-${Stage}'
  - Name: AIRFLOW__LOGGING__REMOTE_LOGGING
    Value: 'true'
  - Name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
    Value: !Sub "cloudwatch://arn:aws:logs:eu-central-1:XXXXXXX:log-group:airflow-v2-${Stage}-dags"
  - Name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
    Value: logs_cloudwatch

    Adding logs:CreateLogGroup to the task role seems to resolve the issue. But why is that necessary?

  13. eladkal commented on Jul 7, 2022

    @eladkal
    eladkal
    on Jul 7, 2022
    Contributor

    Based on discussion in #19022 i'm converting this to github discussion

  14. apache
    locked and limited conversation to collaborators on Jul 7, 2022
  15. eladkal
    converted this issue into a discussion #24892 on Jul 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:coregood first issuekind:bugThis is a clearly a bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions