Skip to content

Add deferrable mode to dataflow operators - #27776

Merged
potiuk merged 3 commits into
apache:mainfrom
lwyszomi:dataflow_defferable_mode
Jan 30, 2023
Merged

potiuk merged 3 commits into
apache:mainfrom
lwyszomi:dataflow_defferable_mode

Conversation

@MrGeorgeOwl

@MrGeorgeOwl MrGeorgeOwl commented Nov 18, 2022 •

Copy link
Copy Markdown
Contributor

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Nov 18, 2022
Comment thread airflow/providers/google/cloud/example_dags/example_dataflow.py Outdated
@MrGeorgeOwl
MrGeorgeOwl force-pushed the dataflow_defferable_mode branch 5 times, most recently from e12f828 to 7595951 Compare November 24, 2022 11:35
@potiuk

potiuk commented Dec 3, 2022

Copy link
Copy Markdown
Member

Tests failing.

@MrGeorgeOwl
MrGeorgeOwl force-pushed the dataflow_defferable_mode branch 3 times, most recently from 079ac53 to 067b705 Compare December 9, 2022 08:42
@MrGeorgeOwl
MrGeorgeOwl force-pushed the dataflow_defferable_mode branch 5 times, most recently from 7b31c11 to 901352d Compare December 19, 2022 08:23
@MrGeorgeOwl

MrGeorgeOwl commented Dec 19, 2022 •

Copy link
Copy Markdown
Contributor Author

@potiuk, tests that are failing at the moment are those that come with changes from main branch. Have the same issues in several PRs so I think that changes in PR are okay but I might be wrong

@potiuk

potiuk commented Dec 19, 2022

Copy link
Copy Markdown
Member

I think there were intermittent/flaky tests - I re-run them to be sure.

Comment thread airflow/providers/google/cloud/hooks/dataflow.py Outdated
Comment thread airflow/providers/google/cloud/operators/dataflow.py Outdated
Comment thread airflow/providers/google/cloud/operators/dataflow.py Outdated
Comment thread airflow/providers/google/cloud/operators/dataflow.py Outdated
Comment thread airflow/providers/google/cloud/operators/dataflow.py Outdated
Comment thread airflow/providers/google/cloud/operators/dataflow.py Outdated
Comment thread airflow/providers/google/provider.yaml Outdated
Comment thread generated/provider_dependencies.json Outdated
@brucearctor

Copy link
Copy Markdown

@MrGeorgeOwl -- this is great and we are eagerly awaiting this functionality to save us some headaches! Do you want some help getting it over the finish line? Looks like the outstanding reviews are pretty straightforward :-)

Please advise.

@MrGeorgeOwl

Copy link
Copy Markdown
Contributor Author

@brucearctor, thank you for your offer of help I appreciate that. I finished required changes at the moment so PR should be merged after it will be reviewed and approved

@MrGeorgeOwl
MrGeorgeOwl force-pushed the dataflow_defferable_mode branch 15 times, most recently from 9977d08 to 683f62f Compare January 30, 2023 08:03
Comment thread airflow/providers/google/cloud/hooks/dataflow.py Outdated
Comment thread airflow/providers/google/provider.yaml Outdated

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment why we are adding upper-bound limit

@MrGeorgeOwl
MrGeorgeOwl force-pushed the dataflow_defferable_mode branch 2 times, most recently from 7fe1466 to 1430b33 Compare January 30, 2023 13:44
@MrGeorgeOwl

Copy link
Copy Markdown
Contributor Author

@potiuk, I've added comment about upper-bound for google library

@MrGeorgeOwl
MrGeorgeOwl force-pushed the dataflow_defferable_mode branch from 1430b33 to e9ede1f Compare January 30, 2023 13:48
@potiuk
potiuk merged commit 094d6bf into apache:main Jan 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants