Repository navigation
Explore the possibility of generic SqlToGcsOperator #21599
Description
Body
Recently SqlToS3Operator was added in PR #20807 (It also deprecated MySqlToS3Operator)
In Google provider Currently we have BaseSQLToGCSOperator with PrestoToGCSOperator, MSSQLToGCSOperator, MySQLToGCSOperator, OracleToGCSOperator, PostgresToGCSOperator, TrinoToGCSOperator, DummySQLToGCSOperator
This task is to check if we can have a generic SqlToGCSOperator just like there is for AWS so we won't need class per each provider...
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
kadai0308 commentedon Feb 20, 2022
Hi eladkal, Can I take this one as my first issue? I already went through the PR #20807
and later I will check each implementation of XXXToGCSOperator to see is it possible to implement a generic
SqlToGCSOperatorto replace all of them.eladkal commentedon Feb 20, 2022
sure assigned to you
kadai0308 commentedon Feb 21, 2022
Hi @eladkal, when I investigated the presto_to_gcs operator & trino_to_gcs operator found the key of integer & float in their type map is different with others to_gcs operator, I think it might be a bug, so raise a PR #21700.
Can you help me check on that? Thanks