GoogleAdsToGcsOperator sends default headers which is no longer supported by the API

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

10.21.1

Apache Airflow version

Version 2.0

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Docker-Compose

Deployment details

Pretty standard docker-compose up. In the docker compose mounted the volumes to the standard directories and set some variables to be loaded into xcoms (eg user, gcs_conn_id). Nothing's customised otherwise.

What happened

I wanted to test GoogleAdsToGcsOperator and I wrote a GAQL query to hit the customer_client resource and the task failed with the following error:

Fault: errors {
  error_code {
    request_error: PAGE_SIZE_NOT_SUPPORTED
  }
  message: "Setting the page size is not supported. Search Responses will have fixed page size of \'10000\' rows."
}

I did not specify any page size (this is initialised here)

This was the task I ran in my DAG:

# I get these from xcoms. It's loaded in with docker-compose
DEFAULT_ARGS = {
    # connections
    'gcp_conn_id': 'google_cloud_default', 
    'google_ads_conn_id': 'google_ads_conn',
    'retries': 2
}

GoogleAdsToGcsOperator(
        task_id='get_google_ads_sub_accounts',
        bucket=BUCKET,
        obj=file_path,
        api_version=API_VERSION, # v17
        gzip=False,
        # default_args=DEFAULT_ARGS,
        client_ids=[MCC_ACCOUNT_ID], # MCC accountID
        attributes=[
            "customer_client.descriptive_name"
            , "customer_client.id"
            , "customer_client.manager"
            , "customer.id"
            , "customer_client.hidden"
            , "customer_client.status"
        ],
        query="""
            SELECT customer_client.descriptive_name
                 , customer_client.id
                 , customer_client.manager
                 , customer.id
                 , customer_client.hidden
                 , customer_client.status
            FROM customer_client
            WHERE customer_client.status NOT IN ('CANCELED', 'CLOSED')
            """
    )

What you think should happen instead

a file should be created in my GCS bucket with the request response.

How to reproduce

If you run the following task with your connection ID information you would probably get the same error. Do not set the page_size parameter so that the init func passes the default.

Anything else

The entire request response headers are:

Headers: {
  "google.ads.googleads.v17.errors.googleadsfailure-bin": "\ni\n\u0002\b(\u0012cSetting the page size is not supported. Search Responses will have fixed page size of '10000' rows.\u0012\u0016kUF6eMWJI6-xDnmxB7Dhww",
  "grpc-status-details-bin": "\b\u0003\u0012%Request contains an invalid argument.\u001a\u0001\nDtype.googleapis.com/google.ads.googleads.v17.errors.GoogleAdsFailure\u0012\u0001\ni\n\u0002\b(\u0012cSetting the page size is not supported. Search Responses will have fixed page size of '10000' rows.\u0012\u0016kUF6eMWJI6-xDnmxB7Dhww",
  "request-id": ""
}
Fault: errors {
  error_code {
    request_error: PAGE_SIZE_NOT_SUPPORTED
  }
  message: "Setting the page size is not supported. Search Responses will have fixed page size of \'10000\' rows."
}

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@shervinrad100

@shervinrad100
Yes, indeed, this was changed in the last release of the API. I saw you want to commit to submitting a PR. I can assign it to you if you want. My only nit is to be careful with older API versions, as the code should be backwards compatible with all versions.

Note: Setting a page_size in version v17 of the Google Ads API will return an error. The earlier versions will change start throwing errors on August 12, 2024. This field will be removed in a future version of the Google Ads API. See the announcement.

https://developers.google.com/google-ads/api/docs/reporting/paging

In the documents, it seems like they will drop this for earlier versions as well. Never mind about the backward compatibility part. I think they are already deprecating older versions of their API.
https://developers.google.com/google-ads/api/docs/sunset-dates

I saw you want to commit to submitting a PR. I can assign it to you if you want

Yes please would love to contribute and this seems like a quick fix anyway.

I was thinking of simply removing the default assignment. If you want backward compatibility then I can add the following to the init:

def __init__(...
...
self.api_version = api_version
if self.api_version < 17:
    self.page_size = page_size or 10000
    # warning message on deprecation. something like:
    # warnings.warn("Setting a page_size in version v17 of the Google Ads API will return an error. The earlier versions will change start throwing errors on August 12, 2024.")

I was thinking of simply removing the default assignment.

This is already done and (just) merged via #43474

@potiuk

@potiuk I might be wrong here but I believe #43474 would force us to declare the API version but would still assign page_size as default which can cause the error.

I believe the page_size default assignment should also be removed.

This is already done and (just) merged via #43474

I think they deprecated this part where we are passing the page_size in the API call. ads_to_gcs.py

If you want backward compatibility then I can add the following to the init:

I think we already passed the 12 of August 2024 so I don't think we should support any backward compatibility in this case. This should already deprecated for all the APIs. Maybe we don't even need to pass any page_size at all.

@bugraoz93

@bugraoz93 If you assign the issue to me I'd be happy to push a PR today

Feel free.

ready for your review ✅

This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.

This issue has been closed because it has not received response from the issue author.

Navigation Menu

Repository navigation

GoogleAdsToGcsOperator sends default headers which is no longer supported by the API #43486

Apache Airflow Provider(s)

Versions of Apache Airflow Providers

Apache Airflow version

Operating System

Deployment

Deployment details

What happened

What you think should happen instead

How to reproduce

Anything else

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

GoogleAdsToGcsOperator sends default headers which is no longer supported by the API #43486

Description

Apache Airflow Provider(s)

Versions of Apache Airflow Providers

Apache Airflow version

Operating System

Deployment

Deployment details

What happened

What you think should happen instead

How to reproduce

Anything else

Are you willing to submit PR?

Code of Conduct

Activity

boring-cyborg commented on Oct 29, 2024

bugraoz93 commented on Oct 29, 2024

bugraoz93 commented on Oct 29, 2024

shervinrad100 commented on Oct 29, 2024

potiuk commented on Oct 29, 2024

shervinrad100 commented on Oct 29, 2024

bugraoz93 commented on Oct 29, 2024

bugraoz93 commented on Oct 29, 2024

shervinrad100 commented on Oct 30, 2024

potiuk commented on Oct 30, 2024

shervinrad100 commented on Oct 30, 2024

github-actions commented on Oct 31, 2025

github-actions commented on Nov 30, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions