Skip to content

Full Refresh arg in invocation_config does not work via DataformCreateWorkflowInvocationOperator #53843

Description

@rsenseman

I'm trying to run Dataform in "Full Refresh" mode with this Airflow Operator definition and it straight up doesn't work. It only runs the repo in non-full refresh mode. This leaves us with effectively no way to run our production DAG via "Full Refresh". I tried sorting through the code but I can't see where the arg is getting dropped.

        create_workflow_invocation_full_refresh = DataformCreateWorkflowInvocationOperator(
            task_id="create_workflow_invocation_full_refresh",
            project_id=PROJECT_ID,
            region=DATAFORM_REGION,
            repository_id=REPOSITORY_ID,
            
            workflow_invocation={
                "compilation_result": "{{ task_instance.xcom_pull('create_compilation_result')['name'] }}",
                "invocation_config": {
                    "fully_refresh_incremental_tables_enabled": True,
                }
            }
        )

Class Definition:

class DataformCreateWorkflowInvocationOperator(GoogleCloudBaseOperator):

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions