Repository navigation
log_query of AthenaOperator dumps un readable query #34189
Description
Body
Log:
Compare to render tab:
Reproduce steps:
set file.sql as:
CREATE EXTERNAL TABLE IF NOT EXISTS `outputs3`(
`region` string,
....
)
set operator as:
AthenaOperator(
task_id="my_task",
query="file.sql",
database="my_db",
log_query=True,
output_location="s3://...",
)
The needed fix:
Pretty print the query. It should appear exactly like it is in the .sql file.
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
mingdaoy commentedon Sep 9, 2023
@eladkal Can I take ownership of this issue? I wanna learn and be curious
eladkal commentedon Sep 9, 2023
Assigned to you @mingdaoy
mingdaoy commentedon Sep 12, 2023
My initial investigation would be format the
paramswithsqlparseand I need to figure out how to use Athena within Airflow to verify the result. Am I on the right track?self.log.info("Running Query with params: %s", params)https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/hooks/athena.py#L118C60-L118C60
sqlparse.format(query, reindent=True, keyword_case='upper')eladkal commentedon Sep 12, 2023
You can setup aws connection with Athena table to test it
I think AWS offers free trail account if you don't have one? cc @shubham22
eladkal commentedon Sep 12, 2023
By the way the
log_queryis useful also for other cases. So we can also add it inBaseSQLOperatorthen you can test it with almost any databasse usingSQLExecuteQueryOperator. Since this is just parsing strings I think it's safe to say that if it works forSQLExecuteQueryOperatorit will work also for Athena.Then open PR for Athena and PR for common.sql